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1  Executive  Summary 

This  guide  provides  information  on  implementing  technieal  measurement  on  a  project. 

Technical  measurement  includes  Measures  of  Effectiveness  (MOEs),  Key  Performance 
Parameters  (KPPs),  Measures  of  Performance  (MOPs),  and/or  Technical  Performance  Measures 
(TPMs).  The  following  are  short  definitions  of  these  terms.  They  are  further  defined  in  the 
Section  3  of  this  guide. 

•  MOEs  are  “operational”  measures  of  success  that  are  closely  related  to  the  achievement 
of  mission  or  operational  objectives;  i.e.,  they  provide  insight  into  the  accomplishment  of 
the  mission  needs  independent  of  the  chosen  solution 

•  MOPs  characterize  the  physical  or  functional  attributes  relating  to  the  system  operation; 
i.e.,  they  provide  insight  into  the  performance  of  the  specific  system 

•  TPMs  measure  attributes  of  a  system  element  within  the  system  to  determine  how  well 
the  system  or  system  element  is  satisfying  specified  requirements 

•  KPPs  are  a  critical  subset  of  the  performance  parameters  representing  the  most  critical 
capabilities  and  characteristics 

Technical  measurement  is  the  set  of  measurement  activities  used  to  provide  the  supplier  and/or 
acquirer  insight  into  progress  in  the  definition  and  development  of  the  technical  solution, 
ongoing  assessment  of  the  associated  risks  and  issues,  and  the  likelihood  of  meeting  the  critical 
objectives  of  the  acquirer.  This  insight  helps  project  management  make  better  decisions 
throughout  the  life  cycle  to  increase  the  probability  of  delivering  a  technical  solution  that  meets 
both  the  specified  requirements  and  the  mission  needs.  The  insight  is  also  used  in  trade-off 
decisions  when  performance  exceeds  the  threshold.  Technical  measurement  is  planned  early  in 
the  life  cycle  and  then  performed  with  increasing  levels  of  fidelity  as  the  technical  solution  is 
developed. 

These  technical  measures  are  periodically  tracked  and  reviewed  by  decision  makers  throughout 
the  lifecycle  to  provide  insight  that  enables  evaluation  and  management  of  technical  progress  and 
risks.  More  specifically,  they  are  used  as  indicators  of  the  following: 

•  Insight  into  likelihood  of  achieving  the  operational  objectives  or  capabilities  that  the 
acquirer  is  expecting 

•  Assessment  of  technical  solution  progress  towards  providing  the  specified  technical 
solution 

•  Assessment  of  how  well  the  technical  solution  complies  with  the  performance 
requirements  per  established  plans 

•  Evaluation  of  the  technical  risk  as  the  solution  evolves 

This  guide  describes  how  technical  measurement  can  be  applied,  using  the  measurement  process 
described  in  Practical  Software  and  Systems  Measurement.  Eessons  learned  in  the  areas  of 
establishing  commitment,  planning,  and  performing  measurement  are  identified.  In  addition, 
candidate  technical  measures  that  are  commonly  used  in  industry  are  identified.  Eigure  1-1 
illustrates  the  relationship  between  the  various  types  of  technical  measures. 
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Figure  1-1  Relationship  of  the  Technical  Measures 


The  intended  audience  for  this  guide  is  the  acquirer  and  supplier  program  and  technical 
management,  as  well  as  the  technical  measurement  professionals  who  must  implement  technical 
measurement  on  their  programs.  This  guidance  is  intended  to  provide  guidance  and  lessons 
learned  on  implementing  technical  measurement,  as  well  as  sample  measures  that  are  commonly 
used  in  industry  today. 

This  guidance  is  the  result  of  a  joint  project  that  was  conducted  between  the  Practical  Software 
and  Systems  Measurement  (PSM)  project,  the  International  Council  On  Systems  Engineering 
(INCOSE),  and  various  companies  including  Lockheed  Martin.  All  information  is  intended  to 
reflect  actual  proven  practice.  Thus,  a  major  source  of  information  for  this  guide  was  the  set  of 
responses  to  a  questionnaire  distributed  to  a  broad  portion  of  the  systems  engineering 
community. 
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2  Introduction 


2. 1  Background 

This  guidance  is  the  result  of  a  joint  project  that  was  conducted  between  the  Praetical  Software 
and  Systems  Measurement  (PSM)  projeet  offiee,  the  International  Couneil  On  Systems 
Engineering  (INCOSE),  and  various  eompanies.  It  eaptures  an  industry  proven  method  to 
aecomplish  technieal  measurement.  The  projeet  was  an  outgrowth  of  joint  workshops  between 
PSM  and  INCOSE  during  the  past  few  years  and  some  internal  work  by  Eoekheed  Martin.  The 
approaeh  for  the  projeet  was  to  leverage  existing  measurement  guidanee  and  implementation 
materials  to  avoid  repeating  basie  measurement  guidanee  or  re-inventing  the  wheel.  In  addition 
to  leveraging  the  literature  eurrently  available  on  this  subject,  this  project  had  an  objective  to 
identify  what  methods  are  being  used  aeross  industry.  This  was  done  through  the  administration 
of  questionnaires  to  a  broad  range  of  engineers.  More  information  on  the  demographies  of 
questionnaire  respondents  is  provided  in  seetion  12.  This  questionnaire  provides  eonfirmation  of 
the  usage  of  teehnieal  measurement  that  is  deseribed  in  this  guide. 

2.2  Objectives 

The  objeetives  of  the  Teehnieal  Measurement  Guide  are  to  ereate  guidanee  on  teehnieal 
measurement  that: 

•  Establishes  guidanee  that  refleets  state-of-the-praetiee  in  industry 

•  Establishes  lessons  learned  aeross  industry  -  i.e.,  what  are  the  proven  methods 

•  Provides  a  consistent  approach  to  technical  measurement  for  projeets 

•  Establishes  a  list  of  eommonly  used  measures 

2.3  Organization  Of  The  Guide 

Seetion  3  presents  the  definitions  of  the  teehnieal  measurement  terms  that  are  eommonly  used  in 
government  or  industry  and  a  general  deseription  of  their  applieation.  A  major  souree  of 
information  for  this  guide  was  the  responses  to  a  questionnaire  distributed  to  a  broad  portion  of 
the  systems  engineering  eommunity.  The  basie  measurement  proeess  defined  in  PSM  is  the 
basis  of  the  measurement  proeess  for  this  guide.  Information  on  the  PSM  measurement  proeess 
ean  be  found  at  http://www.psmso.oom.  This  guide  will  provide  tailoring  guidanee  for 
applieation  to  teehnieal  measurement  in  seotions  5,  6,  and  7.  Seetion  5  discusses  the 
establishment  of  the  oommitment  needed  for  teehnieal  measurement,  Seetion  6  discusses 
planning  the  teehnieal  measurement,  and  Section  7  discusses  performing  the  teehnieal 
measurement.  The  remaining  seotions  of  the  guide  inolude  additional  information  to  assist  the 
planning  and  implementation  of  the  teehnieal  measurement.  Seetion  8  oontains  a  TPM 
cheoklist.  Section  9  discusses  implementation  for  Integrated  Produot  Teams  (IPTs),  Section  10 
contains  a  matrix  of  some  candidate  teehnieal  measures,  and  Seetion  1 1  discusses  the  use  of 
teohnology  readiness  levels.  The  demographies  of  the  survey  respondents  are  provided  in 
seetion  12.  Seetion  13  provides  a  eomprehensive  example  that  explains  the  relationships  among 
MOEs,  MOPs,  and  TPMs.  Section  14  contains  a  list  of  references  that  are  relevant  to  this  guide 
and  Seetion  15  includes  a  list  of  Aeronyms. 
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3  Description  Of  Technicai  Measurement  And  Types  Of 
Technicai  Measures 

3. 1  What  is  Technical  Measurement? 

Technical  measurement  is  the  set  of  measurement  activities  used  to  provide  the  supplier  and/or 
acquirer  insight  into  progress  in  the  definition  and  development  of  the  technical  solution  and  the 
associated  risks  and  issues.  This  insight  helps  project  management  make  better  decisions 
throughout  the  life  cycle  to  increase  the  probability  of  delivering  a  technical  solution  that  meets 
both  the  specified  requirements  and  the  mission  needs.  The  insight  is  also  used  in  trade-off 
decisions  when  performance  exceeds  the  threshold.  Technical  measurement  is  planned  early  in 
the  life  cycle  and  then  performed  with  increasing  levels  of  fidelity  as  the  technical  solution  is 
developed. 

3.2  What  are  the  Types  of  Technical  Measures? 

This  section  defines  and  describes  the  types  of  technical  measures  that  are  commonly  used  in 
government  and  industry  for  insight  into  the  performance  of  the  technical  solution.  Although 
they  may  not  be  used  universally,  they  were  found  to  be  used  widely  by  questionnaire 
respondents.  Measurement  definitions  are  derived  from  leading  sources  to  provide  a  clear  and 
comprehensive  set  of  definitions. 

3.2.1  Measures  of  Effectiveness  (MOEs) 

Definition:  The  “operational”  measures  of  success  that  are  closely  related  to  the  achievement  of 
the  mission  or  operational  objective  being  evaluated,  in  the  intended  operational 
environment  under  a  specified  set  of  conditions;  i.e.,  how  well  the  solution  achieves  the 
intended  purpose.  (Adapted  from  DoD  5000.2,  DAU,  and  fNCOSE) 

MOEs,  which  are  stated  from  the  acquirer  (customer/user)  viewpoint,  are  the  acquirer’s  key 
indicators  of  achieving  the  mission  needs  for  performance,  suitability,  and  affordability  across 
the  life  cycle.  Although  they  are  independent  of  any  particular  solution,  MOEs  are  the  overall 
operational  success  criteria  (e.g.,  mission  performance,  safety,  operability,  operational 
availability,  etc.)  to  be  used  by  the  acquirer  for  the  delivered  system,  services,  and/or  processes. 

MOEs  focus  on  the  system’s  capability  to  achieve  mission  success  within  the  total  operational 
environment.  MOEs  represent  the  acquirer’s  most  important  evaluation  and  acceptance  criteria 
against  which  the  quality  of  a  solution  is  assessed.  They  are  specific  properties  that  any 
alternative  technical  solution  must  exhibit  to  be  acceptable  to  the  acquirer  (i.e.,  the  Standard  of 
Acceptance),  fn  addition  to  using  MOEs  to  compare  and  evaluate  alternatives,  they  can  also  be 
used  for  sensitivity  analysis  of  performance  from  variations  of  key  assumptions  and  parameters 
of  the  potential  alternatives.  They  are  also  important  for  test  and  evaluation  because  they 
determine  how  test  results  will  be  judged.  Since  test  planning  is  directed  toward  obtaining  these 
measures,  it  is  important  that  they  be  defined  early. 

MOEs  are  used  to; 

•  Compare  operational  alternatives 

•  investigate  performance  sensitivities  to  changes  in  assumptions  from  the  user’s  view 


27  December  2005 


9 


Technical  Measurement 


INCOSE-TP-2003-020-01 


•  Define  operational  requirement  values 

•  Evaluate  aehievement  of  key  operational  performance 

•  Serve  as  the  Standard  of  Acceptance  for  the  technical  solution 

3.2.2  Measures  of  Performance  (MOPs) 

Definition:  The  measures  that  characterize  physical  or  functional  attributes  relating  to  the  system 
operation,  measured  or  estimated  under  specified  testing  and/or  operational  environment 
conditions.  (Adapted  from  DoD  5000.2,  DAU,  INCOSE,  and  EPI  280-04,  EM  Integrated 
Measurement  Guidebook) 

MOPs  measure  attributes  considered  as  important  to  ensure  that  the  system  has  the  capability  to 
achieve  operational  objectives.  MOPs  are  used  to  assess  whether  the  system  meets  design  or 
performance  requirements  that  are  necessary  to  satisfy  the  Measures  of  Effectiveness  (MOEs). 
MOPs  should  be  derived  from  or  provide  insight  for  MOEs  or  other  user  needs.  The  relationship 
between  MOEs  and  MOPs  is  illustrated  in  section  3.2.6.  MOPs  are  derived  from  the  supplier’s 
viewpoint  and  look  at  how  well  the  delivered  system  performs  or  is  expected  to  perform  against 
system  level  requirements.  They  address  an  aspect  of  the  system  performance  or  capability. 
MOPs  often  map  to  Key  Performance  Parameters  (KPPs)  or  requirements  in  the  system 
specification.  They  are  expressed  in  terms  of  distinctly  quantifiable  performance  features,  such 
as  speed,  payload,  range,  or  frequency.  They  are  progressively  monitored  and  used  during 
project  execution  as  input  to  management,  including  as  indicators  to  aid  managing  technical 
risks. 

MOPs  are  used  to: 

•  Compare  alternatives  to  quantify  technical  or  performance  requirements  as  derived  from 
MOEs 

>  Support  assessment  of  system  design  alternatives 

>  Support  assessment  of  technical  impact  of  proposed  system  change  alternatives 

•  Investigate  performance  sensitivities  to  changes  in  assumptions  from  the  technical  view 

•  Refine  KPP  definitions 

•  Assess  achievement  KPPs 

This  guide  treats  Measures  of  Suitability  (MOS),  as  a  type  of  MOP  and  thus  has  not  included 
separate  guidance.  The  MOS  specifically  measures  the  extent  to  which  the  technical  solution 
will  integrate  into  the  operational  environment.  As  such,  they  are  often  focused  on  the  usability 
and  interoperability  aspects  of  the  system,  but  may  also  include  other  quality  factors.  In  some 
cases,  it  may  be  necessary  to  define  and  track  MOSs  separately  from  the  MOPs. 

3.2.3  Technical  Performance  Measures  (TPMs) 

Definition:  TPMs  measure  attributes  of  a  system  element  to  determine  how  well  a  system  or 
system  element  is  satisfying  or  expected  to  satisfy  a  technical  requirement  or  goal. 

These  measures  are  used  to  assess  design  progress,  compliance  to  performance  requirements,  or 
technical  risks.  TPMs  are  derived  from  or  provide  insight  for  the  MOPs  focusing  on  the  critical 
technical  parameters  of  specific  architectural  elements  of  the  system  as  it  is  designed  and 
implemented.  The  relationship  between  TPMs  and  MOPs  is  illustrated  in  section  3.2.6. 
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Selection  of  TPMs  should  be  limited  to  critical  technical  thresholds  or  parameters  that,  if  not 
met,  put  the  project  at  cost,  schedule,  or  performance  risk.  The  TPMs  are  not  a  full  listing  of  the 
requirements  of  the  system  or  system  element. 

TPMs  include  the  projected  performance,  such  as  a  performance  profile  with  tolerance  bands  of 
acceptable  variance.  Performance  of  the  system  or  system  element  is  tracked  through  the  life 
cycle  and  compared  to  the  projected  and  required  values.  Early  in  the  life  cycle  the  performance 
values  may  be  estimated,  based  on  simulation  and  modeling.  As  the  life  cycle  proceeds,  actual 
data  replaces  estimates  and  adds  to  the  fidelity  of  the  information.  This  measurement  of  the 
design  solution  as  it  evolves  allows  action  to  be  taken  early  in  the  process,  rather  than  wait  until 
system  testing  to  address  performance  problems.  TPMs  enable  an  assessment  of  the  product 
design  by  estimating  the  values  of  key  performance  parameters  of  the  design  through 
engineering  analyses  and  tests.  Analysis  of  these  measures  provides  risk  indicators  for  key 
performance  parameters. 

TPMs  can  include,  but  are  not  limited  to,  range,  accuracy,  weight,  size,  power  output,  timing 
(throughput,  response  time,  processing  time,  etc.),  security  requirements,  and  the  product  quality 
characteristics  related  to  critical  operational  requirements  (reliability  figure  of  merit,  failure  rate, 
mean  time  to  failure/repair/restore,  availability,  fault  tolerance,  etc.).  A  matrix  of  some 
candidate  measures  is  provided  in  Table  10-1. 

TPMs  are  used  to: 

•  Forecast  the  values  to  be  achieved  for  key  performance  parameters 

•  Identify  differences  between  actual  versus  planned  performance 

•  Assess  and  predict  progress  towards  achieving  the  key  performance  parameters 

•  Determine  the  impact  of  differences  between  actual  and  planned  performance  on  system 
effectiveness 

•  Provide  early  identification  of  risks  and  detection  or  prediction  of  problems  requiring 
management  attention  (e.g.,  where  negative  margins  exist) 

•  Determine  where  opportunities  exist  to  make  design  trades  to  reduce  overall  risk  (e.g., 
where  positive  margins  exist) 

•  Early  determination  of  where  critical  requirement  fiowdown  to  the  next  level  of  design  is 
inadequate 

•  Support  assessment  of  system  element  design  alternatives  or  impacts  of  proposed  change 
alternatives 

•  Monitors  incorporation  and  results  of  new  critical  technologies 

3.2.4  Key  Performance  Parameters  (KPPs) 

Definition:  A  critical  subset  of  the  performance  parameters  representing  those  capabilities  and 
characteristics  so  significant  that  failure  to  meet  the  threshold  value  of  performance  can  be 
cause  for  the  concept  or  system  selected  to  be  reevaluated  or  the  project  to  be  reassessed  or 
terminated.  (Adapted  from  Glossary  of  Defense  Acquisition  Acronyms  and  Terms,  Defense 
Acquisition  University  Press,  January  2001) 

Each  KPP  has  a  threshold  and  objective  value.  KPPs  are  the  minimum  number  of  performance 
parameters  needed  to  characterize  the  major  drivers  of  operational  performance,  supportability, 
and  interoperability.  The  KPPs  represent  the  critical  performance  requirements  (that  are  a  part  of 
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the  Critical  To  Customer  (CTC)  requirements)  and  collectively  characterize  the  overall 
performance  in  summary  form.  They  allow  the  stakeholders  to  evaluate  architectural  and  top- 
level  design  decisions  against  what  is  considered  to  be  most  important. 

The  acquirer  (or  designee)  defines  the  KPPs  at  the  time  the  operational  concepts  and 
requirements  are  defined.  Some  of  the  MOPs  are  usually  selected  to  provide  insight  into  the 
achievement  of  KPPs. 

The  following  questions  may  be  useful  in  determining  whether  a  parameter  should  be  a  KPP; 

•  Is  it  essential  for  defining  the  required  capabilities? 

•  Does  it  contribute  to  significant  improvement  in  the  operational  capabilities  of  the 
enterprise? 

•  Is  it  achievable  and  affordable? 

•  Is  it  measurable  and  testable/verifiable? 

•  Is  the  attribute  reflected  by  the  KPP  able  to  by  analyzed  throughout  the  life  cycle? 

•  If  not  met,  will  the  sponsor  of  the  project  be  willing  to  cancel  or  significantly  restructure 
the  project? 

Often  the  acquiring  organization  has  defined  mandatory  KPPs  to  be  used  in  specific  situations. 
Some  of  those  are  provided  here: 

•  Interoperability  and  supportability  are  often  required  or  recommended  as  a  KPPs  for 
information  technology  systems,  which  includes  any  equipment  or  interconnected  system 
or  subsystem  of  equipment,  that  is  used  in  the  automatic  acquisition,  storage, 
manipulation,  management,  movement,  control,  display,  switching,  interchange, 
transmission,  or  reception  of  data  or  information. 

•  DoD  requires  a  Net-Ready  KPP  for  all  information  technology  and  national  security 
systems  that  are  used  to  enter,  process,  store,  display,  or  transmit  DoD  information  and 
interface  with  external  systems.  The  Net-Ready  KPP  must  be  specified  to  allow 
evaluation  of  interoperability  and  supportability  throughout  the  system’s  life. 

•  Reliability  is  required  to  be  assessed  as  a  potential  KPP  in  some  branches  of  DoD,  since 
it  has  been  shown  to  have  a  significant  impact  on  mission  effectiveness,  logistics 
effectiveness,  and  life  cycle  costs. 

•  Force  protection  and  survivability  are  required  KPPs  when  a  manned  system  or  system 
designed  to  enhance  personnel  survivability  may  be  employed  in  an  asymmetric  threat 
environment. 

•  Cost  may  also  be  considered  a  KPP. 

3.2.5  Attributes  of  Technical  Measures^ 

Technical  Measurement  is  the  continuing  verification  of  the  degree  of  anticipated  and  actual 
achievement  of  technical  parameters.  Measured  values  that  fall  outside  established  decision 
criteria  (tolerance  bands)  alert  management  to  take  action  or  perform  further  investigation. 
Relevant  terms  and  relationships  are  defined  below  and  illustrated  in  Figure  3-1.  In  many  cases, 
the  terms  are  specific  applications  of  concepts  or  terms  in  PSM.  In  those  instances,  the 
relationship  to  the  PSM  term  or  concept  is  identified  in  parentheses  to  aid  using  the  PSM 
measurement  specification  template  to  define  the  technical  measures. 


*  This  section  is  derived  from  the  INCOSE  Systems  Engineering  Handbook,  Version  2,  section  4.2.6,  the  DSMC 
Teaching  Note,  “Technical  Performance  Measurement”,  Robert  Lightsey,  October  1997,  and  the  PSM  Measurement 
Specification  Template. 
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a.  Achieved-to-Date.  Measured  teehnical  progress  or  estimate  of  progress  plotted  and 
compared  with  the  planned  progress  at  designated  milestone  dates.  Early  on,  achieved-to- 
date  progress  may  be  estimated  through  modeling  &  simulation  or  analysis.  This  measure, 
also  known  as  the  current  actual,  is  usually  a  base  measure,  describing  a  single  attribute 
that  is  obtained  directly  by  a  specified  measurement  method. 

b.  Current  Estimate.  The  value  of  a  technical  parameter  that  is  predicted  to  be  achieved  with 
existing  resources  by  the  End  of  Contract  (EOC).  The  ‘Current  Estimate’  will  generally 
be  a  derived  measure  that  is  calculated  based  on  multiple  values  of  the  ‘Achieved-to-Date  ’ 
measure  through  some  specified  function. 

c.  Milestone.  Point  in  time  when  an  evaluation  of  a  measure  is  accomplished.  Typically, 
evaluations  are  made  to  support  management  and  technical  reviews,  during  significant  test 
events,  and  may  also  occur  at  cost  reporting  intervals. 

d.  Planned  Value  (Target).  Predicted  value  of  the  technical  parameter  for  the  time  of 
measurement  based  on  the  planned  profile. 

e.  Planned  Performance  Profile  (Analysis  Model).  Profile  representing  the  projected  time- 
phased  demonstration  of  a  technical  parameter  requirement.  It  describes  the  underlying 
model  of  expected  behavior  of  the  measures  over  time. 

f  Tolerance  Band  (Decision  Criteria).  Management  alert  limits  placed  on  each  side  of  the 
planned  profile  to  indicate  the  envelope  or  degree  of  variation  allowed.  The  criteria  are 
used  to  trigger  action  or  further  investigation.  Tolerance  bands  are  an  acknowledgement  of 
estimating  error  and  reflect  acceptable  risk  limits  associated  with  achieving  the 
performance  measured  by  the  TPM. 

g.  Threshold.  The  limiting  acceptable  value  of  a  technical  parameter;  usually  a  contractual 
performance  requirement. 

h.  Variance(s).  Two  variances  are  essential.  These  are  derived  measures  as  follows: 

1 .  Demonstrated  Technical  Variance  -  the  difference  between  the  ‘Planned  Value’  and  the 
‘Achieved-to-Date’  (or  demonstrated/measured)  value  at  a  specific  point  in  time. 

2.  Predicted  Technical  Variance  -  the  difference  between  the  ‘Planned  Value’  at  EOC  and 
the  ‘Current  Estimate’  of  the  parameter. 
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TECHNICAL 

PARAMETER 

VALUE 

e.g., 

MTBF 


TIME 

Figure  3-1  Technical  Measurement  Profile  Illustration 


3.2.6  Relationship  Of  MOEs,  MOPs,  TPMs,  and  KPPs 

“The  distinction  between  ‘effectiveness’  and  ‘performance’  shows  that  MOEs  and  MOPs  are 
formulated  from  different  viewpoints.  An  MOE  refers  to  the  effectiveness  of  a  solution  and  is 
independent  of  any  particular  solution;  an  MOP  refers  to  the  actual  performance  of  an  entity 
[selected  solution].  The  MOE  refers  to  the  stakeholders’  intention,  whereas  the  MOP  is 
concerned  with  actual  performance  [of  the  supplier’s  solution],  which  may  be  quite  divorced 
from  the  stakeholders’  intentions.  An  MOE  will  indicate  a  property  which  a  potential  solution 
must  possess  in  order  to  meet  a  need:  an  MOP  will  tell  what  something  is  capable  of  doing  even 

'y 

if  this  is  not  necessarily  what  the  stakeholders  want  it  to  do.”  Thus,  an  MOE  can  be  used  to 
validate  that  the  system  meets  the  users'  intended  needs,  and  an  MOP  can  be  used  to  verify  the 
system  meets  the  users'  stated  requirements.  This  in  turn  enables  the  requirements  to  be  validated 
to  meet  the  users'  intended  needs.  See  section  3.2.7  for  further  details. 

“The  difference  between  ‘effectiveness’  and  ‘performance’  as  applied  to  a  solution  [for  a  given] 
need  is  that  ‘effectiveness’  is  a  quality  of  fitness  for  service  or  of  producing  the  results  for  which 
it  was  intended.  ‘Performance’  is  the  quality  of ‘doing  something’,  and  ‘doing  something’  does 
not  necessarily  indicate  fitness  for  service.”^ 

After  the  solution  alternative  has  been  selected,  the  TPMs  then  provide  a  lower  level  view  of 
specific  aspects  of  the  performance  of  the  solution.  The  lower  level  measures  (MOPs  and  TPMs) 
should  be  defined  with  the  higher  level  measures  (MOEs  and  MOPs)  in  mind  in  order  to  ensure 
the  measures  can  be  aggregated  to  provide  the  necessary  insight  for  system  and  operational 
decisions. 

Eigure  3-2  illustrates  the  relationship  between  these  technical  measures  and  KPPs.  It  shows  the 
concept  of  TPMs  being  derived  from  MOPs,  and  MOPs  from  MOEs.  It  also  shows  that  KPPs  are 
generally  derived  from  (but  not  limited  to)  MOEs  and  are  a  primary  influence  on  the  selection  of 


^  Sproles,  Noel,  “Coming  to  Grips  with  Measures  of  Effectiveness”,  University  of  South  Australia,  July  1998. 
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the  MOPs.  As  you  move  from  MOEs  to  MOPs  and  then  to  TPMs,  the  fidelity  of  the  technieal 
insight  and  ability  to  get  more  frequent  insight  increases.  However,  the  scope  of  the  insight 
continues  to  become  narrower. 


Mission  Needs 
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Operating  issues 
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increasing 
Technic^ 
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(Progress 

&Risk) 


Figure  3-2  Relationship  of  the  Technical  Measures 


As  a  simple  example  of  how  MOEs,  MOPs,  and  TPMs  work  together; 

•  An  MOE  may  be  that  a  data  system  is  needed  that  does  not  fail  when  processing  specific 
mission  critical  functions 

•  The  MOP  could  be  the  derived  requirement  that  the  system  be  able  to  provide  uninterrupted 
computing  for  at  least  100  hours  (although  usually  there  will  be  multiple  MOPs) 

•  The  TPMs  that  are  tracked  may  include  fault  tolerance,  redundancy,  and  failure  rate 

As  a  more  detailed  example,  consider  the  following  notional  scenario  for  satellite  development, 
as  illustrated  in  Eigure  3-3.  In  this  example.  Service  life  is  a  MOE  for  the  satellite.  It  is  also 
identified  in  this  example  as  a  KPP,  since  it  is  essential  to  provide  orbital  corrections  required  for 
the  mission.  Service  Life  is  related  to  the  amount  of  propulsion  capacity  for  making  routine 
orbital  corrections,  the  battery  life,  and  the  life  of  solar  cells.  In  this  example  there  is  a 
requirement  for  a  service  life  of  8  years.  Propulsion  capacity  is  one  of  the  potential  MOPs  that 
provide  insight  into  the  service  life.  Enough  capacity  is  needed  to  perform  four  (4)  corrections 
per  year,  plus  a  de-orbit  operation  that  is  equivalent  to  three  (3)  corrections.  Propulsion  capacity 
is  related  to  the  volume  of  propellant  on-board,  the  satellite  mass,  the  thruster  efficiency,  the 
propellant  energy  per  unit  volume,  and  other  factors.  The  team  wants  to  use  known  thrusters  and 
propellant  so  their  values  are  fixed.  It  is  acceptable  for  the  satellite  mass  to  vary,  but  there  is  a 
limit.  The  project  team  needs  to  track  the  satellite  volume  that  can  be  allocated  for  propellant. 

At  first  it  will  be  small,  but  will  increase  as  the  design  matures,  battery  size  decreases,  more 
efficient  physical  configurations  are  found,  etc.  The  potential  design  changes  represent 
opportunities  that  can  reduce  the  overall  technical  risk  and  increase  the  probability  of  mission 
success.  The  available  volume  for  propellant  needs  to  be  17.5  liters. 
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over  time 

Figure  3-3  Notional  Example  of  Selection  of  MOEs/KPPs,  MOPs,  and  TPMs 

As  illustrated  in  the  teehnieal  indieators  in  Figure  3-4,  the  propellant  tank  eapaeity  has  inereased 
over  time  positively  impaeting  number  of  orbital  correetions  possible  (MOP)  and  extending 
service  life  of  satellite  (MOE).  A  battery  size  and  weight  reduction  in  3Q99  (due  to  new 
technology  and  materials)  increased  number  of  orbital  corrections  immediately  due  to  satellite 
mass  reduction,  but  reallocation  of  battery  volume  for  propellant  into  the  design  was  not  realized 
until  4Q99,  after  valuable  opportunity  analysis  was  performed.  However,  testing  of  thrusters  and 
batteries  indicated  slightly  worse  efficiency  than  planned  and  had  negative  impacts  on  the  orbital 
corrections  MOP,  and  hence,  the  service  life  MOE  in  2Q00  and  4Q00,  respectively.  A  more 
detailed  example  of  the  identification  and  usage  of  the  technical  measures  is  provided  in  Section 
13. 
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Figure  3-4  Notional  Example  of  Monitoring  Technical  Measures  for  Project  Decisions 

3.2.7  Technical  Measures  and  the  “V”  Model 

During  the  course  of  the  system  development,  the  focus  on  which  type  of  technical  measure 
(MOE,  KPP,  MOP,  and  TPM)  applies  to  provide  the  necessary  insight  shifts  based  on  the 
stakeholders  and  level  of  detail  being  addressed  by  the  development  activities  at  that  time. 

Figure  3-5  uses  the  “V”  model  to  illustrate  the  relationship  of  the  measures  to  the  processes.  The 
MOEs  apply  at  the  top  level,  focusing  on  acquirer  needs  and  validation.  MOEs  provide  a 
quantification  of  value  to  aid  in  procurement  justification,  and  at  the  end  of  the  development  to 
quantify  the  system  validation. 

KPPs  are  used  by  the  developer  to  establish  the  key  requirements  necessary  to  achieve  the 
MOEs.  KPPs  are  used  to  establish  the  MOPs,  which  are  measured  as  soon  as  possible  and 
repetitively  throughout  development,  testing  and  evaluation.  MOPs  can  be  used  to  verify  the 
system  meets  the  system  technical  requirements.  This  in  turn  enables  these  requirements  to  be 
validated  to  meet  the  mission  requirements  and  acquirer  needs. 

TPMs  are  a  further  break-down  of  the  MOPs  (or  driven  by  risks)  intended  to  provide  measurable 
and  ongoing  insight  into  the  technical  progress.  They  are  measured  through  analysis,  modeling 
and  simulation,  and  then  test  or  the  appropriate  verification  method.  During  the  progress 
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through  the  development,  periodic  estimations  of  the  MOEs  are  derived  using  the  MOP  and 
TPM  values  to  ensure  the  MOEs  are  likely  to  be  met.  If  not,  appropriate  actions  are  taken. 


Figure  3-5  MOEs,  MOPs,  TPMs  and  the  “V”  Model  of  System  Development 


3.3  other  Terms  Relevant  to  Use  of  This  Guide 

Acquirer  -  the  stakeholder  that  acquires  or  procures  a  product  or  service  from  a  supplier. 
(ISO/IEC  15288:2002) 

NOTE:  Other  terms  eommonly  used  for  an  acquirer  are  buyer,  customer,  purchaser.  The  acquirer  may  at  the  same 
time  be  the  owner,  user  or  operating  organization.  For  this  guidance  acquirer  is  used  in  the  broadest  sense  to  include 
all  of  these  parties. 

Customer  -  the  organization  or  person  that  receives  a  product  (ISO  9000:2000) 

Project  -  an  endeavor  with  defined  start  and  finish  dates  undertaken  to  create  a  product  or 
service  in  accordance  with  specified  resources  and  requirements.  (ISO/IEC  15288:2002) 

NOTE  1 :  A  project  may  be  viewed  as  a  unique  process  comprising  coordinated  and  controlled  activities  and  may  be 
composed  of  activities  from  the  Project  Processes  and  Technical  Processes  defined  in  this  International  Standard. 
NOTE  2:  For  use  in  this  guide,  the  tern  Project  is  used  in  the  broadest  sense  and  includes  both  Projects  and 
Programs. 

Stakeholder  -  a  party  having  a  right,  share  or  claim  in  a  system  or  in  its  possession  of 
characteristics  that  meet  that  party’s  needs  and  expectations  (ISO/IEC  15288:2002) 
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Supplier  -  an  organization  or  an  individual  that  enters  into  an  agreement  with  the  aequirer  for  the 
supply  of  a  product  or  service.  (ISO/IEC  15288:2002) 

User  -  individual  who  or  group  that  benefits  from  a  system  during  its  utilization.  (ISO/IEC 
15288:2002) 

NOTE:  The  role  of  user  and  the  role  of  operator  may  be  vested,  simultaneously  or  sequentially,  in  the  same 
individual  or  organization. 

3.4  Use  and  Application  Of  Technical  Measures 

When  tracked  and  reviewed  at  key  milestones  during  the  lifecycle,  technical  measures  are  good 
indicators  of  the  following: 

•  Operational  Objectives 

•  Technical  Solution  Progress 

•  Compliance  to  Performance  Requirements 

•  Technical  Risk 

The  following  subsections  examine  these  uses  and  discuss  the  type  of  technical  measures  that  are 
applicable. 

3.4.1  Indicators  Of  Operational  Objectives 

These  technical  measures  are  used  to  determine  the  ability  of  the  technical  solution  to  meet 
mission  needs  for  performance,  suitability,  and  affordability  across  the  life  cycle  within  the 
intended  operational  environment.  MOEs  look  at  this  from  the  acquirer  perspective,  whereas 
MOPs,  which  are  derived  from  MOEs,  look  at  this  from  the  supplier  perspective.  The  MOEs 
characterize  customer  satisfaction  with  the  performance  of  the  technical  solution.  The  MOPs 
characterize  technical  attributes  relating  to  the  specified  mission  (operational)  requirements  of 
the  technical  solution.  These  attributes  are  those  considered  to  be  key  towards  ensuring  that  the 
system  has  the  capability  to  achieve  mission  success. 

3.4.2  Indicators  Of  Technical  Solution  Progress 

These  measures  focus  on  key  attributes  of  the  performance,  design,  manufacturing,  and 
maintenance.  One  purpose  of  measuring  and  tracking  these  attributes  is  to  ensure  progress 
toward  the  end  goal  of  providing  a  system  that  meets  the  user’s  requirements.  They  can  be 
tracked  as  the  development  and  deployment  of  the  technical  solution  evolve  to  provide  early 
indications  of  when  the  progress  is  not  being  achieved  as  needed  to  meet  key  milestones.  TPMs 
are  the  primary  measures  used  to  provide  this  insight  as  the  technical  solution  evolves. 

3.4.3  Indicators  Of  Compliance  To  Performance  Requirements 

Since  MOPs  and  TPMs  measure  the  technical  attributes  related  to  the  specified  mission  and 
technical  requirements,  when  tracked  over  time,  they  can  serve  as  indicators  of  compliance  to  the 
performance  requirements  of  the  system  and  system  elements.  TPMs  are  the  primary  measures 
used  to  provide  this  insight  as  the  technical  solution  evolves.  The  results  of  the  TPMs  can  be 
used  to  predict  the  associated  MOP  values  to  provide  insight  into  the  likelihood  of  meeting 
performance  requirements  with  the  delivered  system. 

3.4.4  Indicators  Of  Technical  Risk 

Another  purpose  of  these  measures  is  to  provide  insight  into  technical  risks,  using  that  insight  to 
help  assess  the  risks,  aid  in  the  determination  of  the  risk  treatments,  monitor  the  risk  progress. 
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and  identify  other  teehnieal  risks.  A  teehnieal  risk  is  an  event  in  the  projeet  that  has  a  non-zero 
probability  of  oeeurrenee  and  an  unfavorable  eonsequenee  on  teehnieal  performanee  or  quality 
(i.e.,  the  event  has  an  unaeeeptable  risk  exposure).  Teehnieal  risks  ean  oeeur  at  any  time  in  the 
life  cyele  and  ean  lead  to  a  teehnieal  solution  that  does  not  meet  specified  requirements,  mission 
needs,  or  cost  and  schedule  objectives.  Measures  should  be  derived  from  the  identified  risks  or 
problems  (among  other  drivers).  For  each  measure,  there  should  be  at  least  one  risk  or  problem 
associated  with  it.  Technical  risks  are  tracked  by  these  measures  to  provide  insight  into  progress 
towards  meeting  critical  performance  requirements  as  the  technical  solution  evolves.  They 
provide  an  early  warning  about  deviations  in  key  technical  parameters,  which,  if  not  controlled, 
can  impact  system  success  in  meeting  user  needs. 

As  the  technical  solution  matures,  the  actual  values  to  date  of  these  measures  are  compared  with 
the  plan.  If  the  actual  value  meets  the  plan,  it  is  an  indication  that  the  risk-treatment  plan  (also 
known  as  risk-handling  strategy)  has  been  effective;  if  the  actual  value  does  not  meet  the  planned 
value  (and  is  outside  the  expected  variation),  it  indicates  that  the  plan  may  need  adjustment  or 
that  corrective  action  may  be  warranted. 

The  results  of  the  TPMs  can  be  used  to  derive  or  predict  the  associated  MOP  values  to  provide 
insight  into  technical  risk  of  the  delivered  system.  Generally,  the  results  of  one  or  more  MOPs 
provide  input  for  the  MOEs  that  reflect  satisfaction  of  mission  needs.  From  this  perspective, 
these  measures  are  used  to: 

1)  Provide  visibility  of  actual  versus  planned  performance  to  monitor  performance  risks  or 
identify  other  related  technical  risks, 

2)  Provide  early  detection  or  prediction  of  problems  which  require  management  attention,  and 

3)  Support  assessment  of  the  impact  of  proposed  change  alternatives 

Use  of  these  measures  alerts  management  to  potential  performance  deficiencies  before 
irrevocable  cost  or  schedule  impact  occurs.  Where  a  project  also  has  an  overall  risk  assessment 
program,  technical  measurement  provides  data  for  technical  risk  planning  and  assessment.  Input 
from  the  risk  management  process  will  also  assist  in  determining  parameter  criticality  in  the 
TPM  selection  process. 

TPMs  provide  insight  into  the  progress  of  the  definition  and  development  of  the  technical 
solution  and  specific  associated  risks  (uncertain  events)  or  problems  (whether  certain  to  occur  or 
already  have  occurred).  The  risks  and  problems  are  “management  issues”  that  should  have 
appropriate  handling/corrective  action  plans  generated,  implemented,  and  monitored  in 
accordance  with  the  local  risk  management  and  project  management  processes. 

Project  scheduling  should  incorporate  key  dates  for  retiring  risks  associated  with  TPMs.  If  these 
risks  associated  with  the  TPMs  cannot  be  retired  early,  there  must  be  sufficient  budget  and 
schedule  late  in  the  project  to  deal  with  the  ongoing  risk-treatment  plans  and  problems,  if  they 
arise. 

Because  of  their  inherent  risk  insight,  TPMs  have  high  visibility  on  projects,  especially  in 
technical  reviews.  The  SE  lead  or  SE  IPT  lead  (if  IPTs  are  used)  is  usually  responsible  for 
tracking  and  briefing  status  of  TPMs  at  project  status  meetings,  design  reviews,  and 
risk/opportunity  management  board  meetings. 
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4  Measurement  Process 

This  section  presents  an  overview  of  the  measurement  process  from  Practical  Software  and 
Systems  Measurement  (PSM)  that  is  used  as  the  underlying  measurement  process  for  this  guide. 

For  more  detailed  information  about  the  PSM  measurement  process,  go  to: 
http://www.psmsc.com. 

There  are  three  key  measurement  concepts  that  form  the  basic  building  blocks  for  successful 
measurement  application.  They  are: 

1 .  Measurement  is  a  consistent  but  flexible  process  that  must  be  tailored  to  the  unique 
information  needs  and  characteristics  of  a  particular  project  or  organization.  These  information 
needs  usually  change  during  the  life  cycle  as  the  environment  changes,  milestones  are 
accomplished,  performance  parameters  are  achieved,  risks  are  treated,  etc.  Changing 
information  needs  drive  changes  to  the  measures. 

2.  Decision  makers  must  understand  what  is  being  measured.  Key  decision  makers,  including 
both  technical  and  business  managers,  must  be  able  to  connect  “What  is  Being  Measured”  to 
“What  they  need  to  know”.  Measurement  must  deliver  value-added  objective  results  that  can  be 
trusted  on  the  day-to-day  issues  that  these  managers  face. 

3.  Measurement  must  be  used  to  be  effective.  The  measurement  program  must  play  a  role  in 
helping  decision  makers  understand  project  and  organization  issues  and  to  evaluate  and  make 
key  trade-offs  to  optimize  overall  performance. 

These  three  basic  measurement  concepts  appear  to  be  common  sense,  but  are  often  ignored. 

They  need  to  be  ingrained  in  the  project  and  organization  to  effectively  apply  measurement.  The 
remainder  of  this  section  presents  the  measurement  process  and  discusses  each  of  these  concepts 
in  more  depth. 


4. 1  The  Measurement  Process 

An  underlying  concept  of  measurement  it  that  it  should  be  flexible  and  tailorable  based  on  the  unique 
information  needs  and  characteristics  of  each  project  or  organization.  The  measurement  process  must  be 
applied  iteratively  to  effectively  adapt  to  changing  information  needs  and  improvements  in  the 
measurement  process  itself 

The  PSM  measurement  process  has  been  the  basis  for  establishing  a  common  process  across  the 
software  and  systems  engineering  communities.  The  overall  measurement  approach  has  been 
adopted  by  the  Capability  Maturity  Model®  Integration  (CMMI^'^j  as  a  basis  for  the 
measurement  and  analysis  process  area  and  by  the  international  software  and  systems 
engineering  community,  as  embodied  in  the  international  systems  and  software  engineering 
standard,  ISO/IEC  15939  -  Measurement  Process.  The  measurement  approach  has  also  been 
adopted  by  the  International  Council  on  Systems  Engineering  (INCOSE)  in  the  work  of  their 
Measurement  Working  Group,  including  the  INCOSE  SE  Measurement  Primer.  An  important 
aspect  of  all  these  initiatives  is  the  consistent  treatment  of  measurement. 
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The  PSM  process,  shown  in  Figure  4-1,  describes  four  activities  that  are  part  of  a  successful 
measurement  program: 

1 .  Establish  and  Sustain  Commitment:  This  activity  involves  establishing  the  resources,  training, 
and  tools  to  implement  a  measurement  program  effectively,  and  most  importantly,  ensuring  that 
there  is  management  commitment  to  use  the  information  that  is  produced. 

2.  Plan  Measurement:  In  this  activity,  measures  are  defined  to  provide  insight  into  project  or 
organization  information  needs.  This  includes  identifying  what  the  decision  makers  need  to 
know,  relating  these  information  needs  to  those  entities  that  can  be  measured,  and  then 
identifying,  prioritizing,  selecting  and  specifying  prospective  measures  based  on  project  and 
organization  processes.  The  specification  of  the  measures  provides  documentation  of  the 
information  needs  and  selected  measures  to  establish  a  common  understanding  of  what  is  being 
measured. 

3.  Perform  Measurement:  This  activity  involves  collecting  and  preparing  measurement  data, 
performing  measurement  analysis,  and  presenting  the  results  so  that  the  information  can  be  used 
to  make  decisions.  The  preparation  of  the  measurement  data  includes  verification, 
normalization,  and  aggregation  of  the  data,  as  applicable.  Analysis  includes  estimation, 
feasibility  analysis  of  plans,  and  performance  analysis  of  actual  data  against  plans.  The 
presentation  of  the  results  should  be  in  the  preferred  format  of  the  decision  maker,  in  order  to 
allow  accurate  and  expeditious  interpretation  of  the  results. 

4.  Evaluate  Measurement:  In  this  activity,  both  the  measurement  process  and  the  specific 
measures  should  be  periodically  evaluated  and  improved  as  necessary.  This  activity  addresses 
the  following  questions: 

1)  Is  the  measurement  process  effective  (i.e.,  is  the  information  being  provided  reliably, 
in  a  cost-effective  and  timely  manner,  and  used  by  decision  makers)? 

2)  Are  the  measures  effective  (i.e.,  do  they  provide  the  insight  needed  by  the  decision 
makers)? 

3)  Are  there  opportunities  to  improve  the  process  or  the  measures? 
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Figure  4-1  Four  Key  Activities  of  the  PSM  Measurement 

Process 

A  measurement  proeess  that  is  flexible  and  tailored  to  projeet  and  organizational  proeesses 
ensures  that  measurement  is  cost  effective.  Data  should  not  be  collected  or  reports  distributed 
that  are  not  needed  or  are  not  used.  In  addition,  data  collection  and  reporting  should  be 
automated  whenever  possible  to  provide  an  automatic  by-product  of  normal  project  activity. 

The  process  shown  in  Figure  4-1  provides  a  foundation  for  measurement  for  many  disciplines 
including  software  engineering,  systems  engineering,  project  management,  and  process 
improvement.  An  important  thing  to  remember  is  that  the  same  basic  measurement  process  can 
support  a  wide  variety  of  distinct  and  changing  information  needs  in  each  of  these  areas. 


4.2  Connecting  Information  Needs  to  Actual  Measures 

The  second  basic  concept  of  successful  measurement  is  the  communication  of  meaningful  information 
to  the  decision  makers.  It  is  important  that  the  people  who  use  the  measurement  information  understand 
what  is  being  measured  and  how  it  is  to  be  interpreted. 

PSM  does  this  by  incorporating  a  measurement  information  model  that  links  the  entities  that  are 
measured  to  the  associated  measures  and  ultimately  to  the  identified  information  need.  The 
measurement  information  model  provides  a  structure  for  specifying  how  a  particular  information  need 
will  be  addressed  within  the  measurement  process.  This  allows  the  measures  to  be  clearly  and 
consistently  defined. 
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4.3  Understanding  and  Using  the  Measurement  Resuits 

The  third  concept  of  successful  measurement  is  that  the  measurement  process  is  an  integral  part 
of  the  way  business  (or  project  execution)  is  conducted.  In  a  successful  measurement 
implementation,  the  measurement  results  are  regularly  used  to  make  decisions.  If  the  members 
of  a  project  or  organization  are  not  able  or  willing  to  use  measurement  data  to  make  decisions, 
the  measurement  program  is  of  little  use. 

To  support  the  use  of  measurement,  information  must  be  obtained  early  enough  to  allow 
managers  to  take  the  actions  necessary  to  reduce  risks  or  correct  problems.  Management 
decisions  cannot  wait  for  a  complete  set  of  perfect  data  to  support  management  decisions,  but 
should  be  derived  from  analysis  of  the  best  available  data,  complemented  by  real-time  events, 
and  qualitative  insight  (including  experience). 

The  risk  management  and  measurement  processes  should  always  be  closely  aligned.  Risk 
management  identifies  the  information  needs  that  can  impact  project  and  organizational 
performance  -  information  needs  that  should  be  objectively  explained  with  the  measurement 
results.  The  measurement  data  helps  to  quantify  risks,  and  subsequently  provides  information 
about  whether  risks  have  been  successfully  mitigated. 
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5  Establishing  Commitment  For  Technical  Measurement 

5. 1  General  Practice 

As  is  the  case  with  any  part  of  the  measurement  program,  it  is  essential  to  establish  the 
importance  of  measurement  and  build  commitment  to  support  the  measurement  tasks  throughout 
the  life  cycle.  The  INCOSE  SE  Measurement  Primer  provides  a  good  overview  of  this  topic, 
while  the  PSM  Guidebook  provides  in-depth  discussion.  In  order  to  establish  commitment,  it  is 
necessary  to  identify  the  stakeholders  and  understand  their  interests.  Section  5.2  provides  a 
discussion  of  the  stakeholders  for  technical  measurement.  Whenever  possible,  benefit  is 
obtained  from  establishing  joint  commitment  and  objectives  between  the  stakeholders.  Section 
5.3  provides  a  discussion  of  how  to  work  toward  a  consistent  technical  measurement  program 
among  the  stakeholders. 

5.2  Who  Are  The  Stakeholders? 

The  primary  stakeholders  of  technical  measurement  vary  depending  on  the  type  of  technical 
measure.  Table  5-1  shows  the  primary  stakeholders  by  types  of  technical  measures,  derived 
from  the  questionnaire  administered  for  this  guide  (see  section  12  for  more  information  on 
questionnaire  respondents).  Since  the  acquirer  is  the  ultimate  stakeholder  of  the  technical 
solution,  the  acquirer  is  obviously  a  stakeholder  of  the  top-level  measures  (MOEs  and  MOPs). 

As  indicated  previously,  the  acquirer  usually  establishes  the  MOEs  and  uses  them  as  part  of 
acceptance  criteria.  The  engineering  staff  of  the  supplier  is  concerned  with  providing  a  technical 
solution  that  achieves  the  MOEs,  but  needs  to  translate  them  into  measures  that  focus  on  the 
technical  aspects  of  the  solution  that  relate  to  performance.  Thus,  they  are  primary  stakeholders 
for  the  MOPs  and  TPMs.  Since  the  IPT  structure  has  members  from  both  the  acquirer  and 
supplier,  the  IPTs  will  be  stakeholders  of  each  of  these  types  of  measures.  The  questionnaires 
indicated  that  quality  management  was  mostly  focused  on  the  MOEs,  since  they  are  responsible 
for  assuring  a  solution  that  will  satisfy  the  acquirer. 


Primary  Stakeholders 

MOE 

MOP 

TPM 

Acquirer  /  Customer 

X 

X 

Engineering  Staff  of  Supplier 

X 

X 

Integrated  Poduct  Team  (IPT) 

X 

X 

X 

Quality  Management 

X 

Note:  Each  listed  item  was  required  to  have  >40%  of  respondents  to  identify  it  as  a  primary  stakeholder. 


Table  5-1  Primary  Technical  Measurement  Stakeholders 


5.3  Establishing  Joint  Objectives  Between  Acquirer  And  Supplier 

It  is  a  good  practice  to  establish  joint  measurement  objectives  and  information  needs  between  the 
acquirer  and  supplier  organizations.  By  working  towards  joint  objectives,  one  can  select  and 
define  common  measures.  This  can  significantly  reduce  the  overall  resources  needed  to  support 
measurement,  while  improving  team  perspectives.  The  following  are  recommended  actions  to 
help  establish  joint  objectives  and  identify  common  information  needs; 
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•  Establish  agreement  on  the  appropriate  level  of  effort  for  the  teehnical  measurement, 
including  responsibilities. 

•  Coordinate  and  prioritize  tasks  for  measurement.  Determine  which  tasks  can  be  jointly 
supported  by  the  acquirer  and  supplier  and  plan  accordingly. 

•  Determine  what  information  is  needed  by  ah  stakeholders  throughout  the  life  cycle. 
Identify  information  that  can  be  addressed  using  the  same  data,  measures,  or  indicators. 
Consider  developing  a  joint  measurement  plan  for  the  measures  that  address  common 
needs. 

•  Maximize  communication  with  the  acquirer  and  IPTs,  if  applicable.  Ensure  time  for 
reviews  with  key  stakeholders.  This  includes  periodically  scheduled  reviews  and  ad  hoc 
reviews,  as  needed.  The  reviews  should  concentrate  on  outliers  and  trends  observed. 

•  Establish  a  common  reporting  format  to  improve  consistency  of  interpretation  by  ah 
stakeholders.  When  a  common  format  can  be  selected,  it  may  be  more  useful  and  cost- 
effective.  The  reports  need  to  include  information  that  both  the  acquirer  and  supplier 
need  for  monitoring  the  project  and  products. 

Parameters  to  be  tracked  are  typically  based  on  the  combined  needs  of  the  acquirer  and  the 
supplier.  The  measures  of  interest  to  the  acquirer  are  those  that  are  focused  on  operational 
needs.  The  supplier  will  generally  track  more  measures  than  are  reported  to  the  acquirer,  as  the 
supplier  needs  more  detailed  information. 
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6  Planning  Technical  Measurement 

6. 1  Determining  And  Prioritizing  information  Needs 

Practical  Software  and  Systems  Measurement  (PSM)  has  a  set  of  seven  information  eategories 
that  address  most  of  the  information  needs  of  a  project.  They  are  used  to  help  foeus  the  seleetion 
of  measures  on  the  set  of  information  needs  for  which  insight  is  essential  for  deeision  support. 
The  teehnieal  measures  are  primarily  found  in  two  of  these,  Product  Size  and  Stability  and 
Product  Quality.  These  eategories  foeus  on  the  physieal  and  quality  attributes  of  the  teehnieal 
solution  that  impaet  performance.  Eaeh  eategory  eontains  “measurable  eoncepts”  that  are  used 
to  help  add  specificity  about  the  information  need.  The  measurable  eoneept  generally  reflects  a 
specific  attribute  of  the  information  need  that  ean  help  prioritize  the  need  and  identify  potential 
measures.  Table  6-1  shows  example  PSM  Categories  and  Coneepts  that  ean  apply  to  MOEs, 
MOPs  and  TPMs.  In  PSM,  the  Product  Size  and  Stability  information  eategory  contains  two 
measurable  eoncepts.  Physical  Size  and  Stability  and  Functional  Size  and  Stability.  Only  the 
Physical  Size  and  Stability  eoneept  is  shown  here,  sinee  the  assoeiated  potential  measures  for 
Functional  Size  and  Stability  are  usually  not  eonsidered  as  MOEs,  MOPs,  or  TPMs.  The  list  of 
associated  potential  measures  does  not  start  to  address  all  possible  measures  related  to 
performanee,  sinee  a  large  majority  of  requirements  eould  translate  into  possible  teehnieal 
measures.  Identifieation  and  selection  of  measures  will  be  discussed  further  in  seetion  6.2,  and  a 
eomplete  list  of  proposed  candidate  measures  is  included  in  section  10. 

PSM-Product-Related-Measurement  Information 


Information  Category 

Measurable  Concept 

Product  Size  and  Stability 

Physical  Size  and  Stability 

Product  Quality 

Functional  Correctness 

Supportability-Maintainability 

Efficiency 

Portability-Usability 

Dependability-Reliability 

Table  6-1  Example  PSM  Information  Categories  and  Measurable  Concepts  Applicable  to 

Tecbnical  Measurement 

Table  5-1  defines  the  primary  stakeholders  who  determine  the  speeific  issues  or  information 
needs  that  are  used  to  identify  potential  measures.  A  diseussion  of  the  stakeholders  and  the  types 
of  teehnieal  measures  are  provided  in  seetion  5.1.  Table  6-2  provides  the  primary  drivers  that 
influence  the  selection  of  information  needs  and  assoeiated  measures.  These  stakeholders  and 
drivers  were  identified  through  analysis  of  the  teehnieal  measurement  questionnaire  assoeiated 
with  this  projeet. 
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Primary  Drivers 

MOE 

MOP 

TPM 

Historical  Operational  Information 

X 

Operational  Risks 

X 

User  Priorities 

X 

Measures  of  Effectiveness 

X 

Key  Performance  Parameters 

X 

Customer  Priorities 

X 

X 

Technical  Requirements 

X 

X 

Technical  Alternatives  Considered 

X 

X 

Measures  of  Performance 

X 

Note:  Each  listed  item  was  required  to  have  >40%  of  respondents  to  identify  it  as  a  primary 
driver,  to  he  included.. 

Table  6-2  Primary  Drivers  for  Information  Need  and  Measure  Identification 

From  Table  6-2,  it  can  be  deduced  that  the  operational  needs  (including  historical  operational 
information,  operational  risks,  and  user  priorities)  and  technical  requirements/  parameters 
(including  MOEs,  KPPs,  MOPs,  and  technical  requirements)  are  the  primary  drivers  for 
identifying  the  information  needs  and  measures  of  the  project.  The  technically  critical  insight 
can  often  be  determined  through  review  of  Systems  Engineering  documentation  (such  as  concept 
of  operation  documents,  operational  requirements  documents,  or  mission  needs  statements), 
mission  or  technical  specification  requirements,  and  planned  contractual  performance  incentives 
and  their  relationship  to  the  technical  measures. 

In  summary,  there  are  several  factors  that  should  be  considered  in  the  prioritization  of  the 
information  needs.  These  include; 

•  Magnitude  of  the  contribution  of  the  item  to  the  overall  performance 

•  Maturity  of  necessary  technologies  that  could  be  a  risk  to  achieving  performance 
objectives 

•  Ability  to  discriminate  among  design  alternatives  or  other  technical  decision  alternatives 

•  Ability  to  handle  technical,  cost  or  schedule  risk  within  management  resources  and 
reserves 

•  Impact  of  risk  on  design  alternatives  and  solution  effectiveness  for  the  user 

•  Ability  to  serve  as  a  Standard  of  Acceptance  for  the  technical  solution 

6.2  Identifying  Potential  Technical  Measures 

Applicable  potential  technical  measures  need  to  be  identified  for  the  prioritized  set  of  specific 
information  needs  (or  issues).  They  are  usually  related  to  the  technical  requirements  or 
parameters  for  which  they  are  providing  insight  into  progress  or  risk.  It  is  important  to 
understand  what  aspect  of  the  technical  requirement  or  measure  is  important  to  monitor;  the 
range  of  performance  for  a  parameter,  the  stability  of  the  parameter,  the  progress  of  a  critical 
technology,  etc.  The  same  factors  that  apply  to  identification  of  valid  potential  measures  in  PSM 
also  apply  here. 

Table  6-1  identifies  the  typical  measurable  concepts  that  may  be  used  to  help  focus  technical 
information  needs  and  identify  applicable  measures.  There  are  numerous  potential  MOEs, 

MOPs  and  TPMs,  since  they  are  related  to  the  performance  requirements  of  the  technical 
solution  and  can  reflect  nearly  any  quantified  product  (or  process)  characteristic.  Table  10-1 
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provides  a  useful  example  set  of  measures  and  provides  insight  into  the  seope  of  applieation  of 
the  measures  aeross  engineering  applieations  or  domains.  Other  potential  measures  ean  be  found 
in  some  of  the  referenees  in  section  12. 

6.3  Selecting  And  Specifying  Technical  Measures 

6.3.1  Selecting  And  Specifying  MOEs 

MOEs  are  usually  determined  by  the  acquirer  (customer/user)  during  the  development  of  the 
mission  needs/requirements  and  operational  concepts,  and  should  be  included  in  the  Request  For 
Proposal  and/or  contract.  Each  MOE  should  provide  insight  into  at  least  one  operational 
objective  or  mission  requirement  of  the  delivered  technical  solution.  Of  the  MOEs,  the  subset 
that  are  so  critical  that  they  must  be  met  to  continue  with  the  program  are  designated  as  Key 
Performance  Parameters  (KPPs).  Some  government  organizations  require  specific  KPPs  for 
various  aspects  of  the  technical  solution  depending  on  the  type  of  system  or  mission 
characteristics.  See  Section  3.2.4  for  some  examples. 

MOEs  should  not  be  strongly  correlated  to  each  other,  in  order  to  provide  insight  into  different 
operational  aspects  of  the  technical  solution  or  solution  alternatives.  The  MOEs  should  be 
defined  in  terms  of  the  acquirer’s  operational  objectives.  The  MOEs  need  to  provide  an 
independent  means  to  collectively  evaluate  the  alternatives  for  their  fitness  to  meet  the  intended 
purpose,  as  well  as  aid  system  validation  through  quantification  of  the  operational  performance. 
Thus,  the  MOEs  are  usually  identified  and  defined  at  the  beginning  of  top-level  analysis  of 
alternatives.  Since  the  MOEs  are  defined  early  in  the  life  cycle,  they  are  usually  based  on  some 
assumptions.  These  assumptions  must  be  validated  with  the  users.  Otherwise,  the  assumptions 
may  be  incorrect  and  lead  to  incorrect  or  infeasible  test  criteria. 

Although  they  may  be  qualitative  or  subjective  (relying  on  expert  judgment),  the  MOEs  are  more 
effective  if  they  are  defined  quantitatively,  including  the  evaluation  criteria  and  data 
requirements  for  each  measure.  They  are  generally  stated  as  raw  quantities  like  numbers  of 
something  or  frequencies  of  occurrence.  Since  MOPs  and  TPMs  are  successively  derived  from 
the  MOEs,  care  should  be  used  to  limit  the  number  of  MOEs  selected  (the  questionnaires  showed 
a  range  of  2  to  12  MOEs  with  an  average  of  6).  It  is  not  unusual  to  have  an  order  of  magnitude 
more  TPMs  than  MOEs. 

6.3.2  Selecting  And  Specifying  MOPs 

MOPs  are  derived  from  the  MOEs  to  provide  measures  based  on  the  technical  requirements  that 
address  the  KPPs  that  trace  to  the  mission  needs.  There  are  generally  several  MOPs  for  each 
MOE  (the  questionnaires  showed  a  range  of  1  to  10  MOPs  per  MOE  with  an  average  of  5).  The 
MOPs  that  trace  to  the  same  MOE  will  be  related  in  the  insight  they  provide,  focusing  on  a 
specific  system  characteristic  or  alternative.  Traceability  should  be  maintained  from  the  MOPs 
to  the  system  level  performance  requirements,  goals,  risks,  or  issues.  The  MOPs  should  also  be 
traceable  to  the  higher  level  MOEs  with  the  relationship  defined,  so  they  can  be  readily  used  to 
provide  insight  into  operational  performance  risks.  It  is  preferable  to  have  a  quantitative 
relationship. 

Collectively,  the  MOPs  should  be  able  to  provide  insight  into  system  affordability,  technical 
performance,  supportability,  suitability,  and  system  level  technical  risk.  Thus,  the  MOPs  should 
be  linked  to  the  system  level  testing  of  alternatives  and  KPPs  to  understand  predicted  or  actual 
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achievement  of  the  performanee  requirements  and  ultimately  the  mission  objeetives.  As  sueh, 
the  MOPs  are  usually  identified  and  defined  during  the  planning  for  test  and  evaluation.  The 
MOPs  should  be  defined  quantitatively,  ineluding  the  evaluation  eriteria  and  data  requirements 
for  each  measure. 

6.3.3  Selecting  TPMs 

TPMs  are  established  early  in  a  projeet  (often  during  the  proposal  stage),  usually  by  the  supplier, 
and  may  be  part  of  the  aequirer’s  mission  requirements.  They  are  usually  derived  from  MOPs, 
however,  a  significant  number  of  respondents  to  the  questionnaire  indicated  direct  derivation 
from  the  MOEs  (or  KPPs).  Generally,  there  is  at  least  one  TPM  per  MOP,  but  often  there  are 
several  TPMs  per  MOP  (the  questionnaires  showed  a  range  of  2-400  TPMs  per  projeet,  with  an 
average  of  54,  and  1-7  TPMs  per  MOP  with  an  average  of  4).  The  TPMs  should  be  selected  and 
defined  sueh  that  they  ean  provide  support  trades  among  possible  solutions  for  aehieving  KPPs. 
These  measures  are  used  to  help  evaluate  the  feasibility  of  the  requirements  set  with  respeet  to 
meeting  the  performance  needs  of  the  user  for  a  given  eandidate  teehnieal  solution  definition, 
while  meeting  other  objeetives.  They  are  strongly  influenced  by  the  quality  requirements.  This 
ongoing  measurement  proeess  allows  for  in-proeess  evaluation  and  timely  eorrective  actions,  if 
needed.  Thus,  the  seleetion  of  TPMs  needs  to  consider  these  faetors  and  uses. 

“Another  important  consideration  is  the  relationship  between  the  TPM  program  and  risk 
management.  Generally,  the  parameters  selected  for  tracking  should  be  related  to  the  risk  areas 
on  the  project.  If  a  particular  element  of  the  design  has  been  identified  as  a  risk  area,  then 
parameters  should  be  seleeted  whieh  will  enable  the  manager  to  traek  progress  in  that  area.  For 
example,  if  aehieving  a  required  aircraft  range  is  eonsidered  to  be  critical  and  a  risk  area,  then 
tracking  parameters  that  provide  insight  into  range  would  be  selected,  such  as  aircraft  weight, 
speeifie  fuel  consumption,  drag,  etc.  Furthermore,  there  should  be  eonsisteney  between  TPMs 
and  the  eritieal  parameters  assoeiated  with  formal  testing,  although  the  TPM  program  will  not 
normally  be  limited  just  to  those  parameters  identified  as  eritieal  for  test  purposes.”^ 

Insight  regarding  eonfidenee  of  aehieving  MOE  or  KPP  objeetives  is  provided  by  MOPs  at  the 
top-level  of  the  teehnieal  solution.  The  MOPs  are  traeeable  to  the  system-level  requirements, 
which  in  turn  are  traceable  to  the  mission  requirements  and  user  needs.  The  lower-level 
parameters  are  identified  through  the  requirements  alloeation  proeess.  These  parameters 
represent  allocation  of  system-level  requirements  to  lower  levels  within  the  system  hierarehy 
and  should  be  available  in  the  doeumentation  of  the  funetional  analysis  proeess.  The  lower-level 
parameters  are  measured  by  TPMs  and  are  traeeable  to  the  MOPs.  The  KPPs  and  their  measures 
should  be  seleeted  using  the  full  scope  of  the  Systems  Engineering  process.  A  eomprehensive  set 
of  key  parameters  should  be  seleeted  for  the  system,  for  eaeh  segment,  and  for  eaeh  eritieal 
configuration  item  (Cl),  on  the  basis  of  overall  teehnieal  importanee,  teehnieal  risk  assessment, 
parametrie  sensitivity  in  the  engineering  models,  and  interfaee  relationships. 

Satisfying  a  TPM  often  involves  multiple  system  elements.  For  example,  the  aggregate  weight 
of  all  the  hardware  eomponents  must  be  less  than  the  system  weight  TPM,  or  the  interfaees, 
proeessors,  software,  and  networks  must  together  satisfy  a  system  throughput  TPM.  Therefore, 
TPMs  must  often  be  budgeted  and  allocated  to  multiple  system  elements,  and  tracking/reporting 
may  be  required  at  this  lower  level  of  granularity.  Alloeated  TPM  budgets  should  be  flowed 
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down,  not  only  to  development  organizations,  but  also  procurement  and  subcontract  management 
organizations,  so  purchased  items  will  satisfy  their  portion  of  TPMs. 

When  TPMs  are  used  in  a  hierarchical  manner  or  TPM  budget  allocations  are  in  place,  a  clear 
roll-up  or  aggregation  methodology  should  be  defined.  Because  tracking  a  TPM  may  be 
complex,  and  involve  multiple  system  elements,  it  may  be  prudent  to  use  prototyping,  simulation 
or  modeling  early  in  a  development  to  demonstrate  that  the  TPM  can  be  satisfied,  and  to  better 
quantify  any  risks. 

The  measures  should  be  selected  based  on  the  following  criteria: 

1 .  They  are  the  most  significant  qualifiers  or  determinants  of  the  total  system  product, 
providing  insight  into  key  performance  parameters  and  mission  success  based  on  one  or 
more  of  the  following: 

•  Represent  an  important  performance  requirement  driving  the  design  of  a  prime  item 
or  a  key  critical  item 

•  Defines  critical  interface  performance  between  system  elements  or  between  a  system 
element  and  the  system  interfaces 

•  Represents  key  functional  capabilities  or  quality  characteristics,  identified  as  having 
moderate  to  high  technical  risks,  that  warrant  continuous  monitoring 

•  Reflects  known  concerns  about  particular  system  issues 

•  Characterizes  and  assesses  performance  risk  of  new  technology  implementation  for 
system  elements 

2.  Are  traceable  to  specification  requirements  or  goals 

3.  Time-phased  values  and  tolerance  bands  can  be  predicted  for  each  parameter  and 
substantiated  during  design,  development,  and  test 

4.  Can  be  measured  on  an  ongoing  basis,  directly  measured  from  tests  or  derived  from 
analyses/models,  allowing  prediction  of  or  insight  into  delivered  performance 

5.  Indicate  key  subcontractor  technical  requirements  and/or  inputs,  when  extended  to 
subcontractors 

6.3.4  Specifying  The  TPMs 

TPMs  should  be  defined  so  they: 

•  Are  traceable  to  the  specified  requirements,  goals,  risks,  or  issues  of  the  system  element 

•  Are  traceable  to  the  higher  level  MOPs  with  relationship  defined 

•  Are  traceable  to  applicable  WBS  elements  (preferred,  but  not  always  practical) 

•  Have  the  ability  to  track  adherence  to  technical  constraints  as  the  technical  processes  are 
performed,  as  well  as  relationships  to  cost,  schedule,  and  quality  objectives 

•  Provide  insight  into  IPT/project  success  on  an  ongoing  basis 

•  Provide  insight  into  system  effectiveness/utility 

Tracking  TPMs  during  the  development  cycle  should  be  an  integral  part  of  a  quantitative 
approach  to  the  successful  management  of  project  goals  and  objectives.  TPMs  should  be  part  of 
the  project’s  measurement  program  with  the  TPMs  documented  in  the  measurement  plan  or  the 
systems  engineering  plan.  A  cost-effective  approach  for  accurate  and  timely  measurement  and 
regular  reporting  of  progress  toward  achieving  TPMs  is  necessary.  It  is  necessary  to  identify  the 
methods  to  be  used  in  measuring  and  assessing  each  parameter.  Analysis  and  reporting  should 
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be  defined  in  a  way  to  provide  an  indieation  of  teehnieal  progress,  potential  future  impaet 
(predietive  indieators),  and  projeet  status  in  a  eomposite,  easy-to-understand  assessment. 
Appropriate  milestones  for  the  TPM  data  eolleetion,  analysis  and  reporting  should  be  defined. 
The  following  diseussion  refers  to  the  elements  of  TPMs  illustrated  in  Figure  3-1. 

The  definitions  of  the  TPMs  should  inelude  appropriate  planned  performanee  profiles,  thresholds 
and  toleranee  bands  (or  risk  bands).  The  use  of  these  meehanisms  provides  a  management  by 
exeeption  tool  that  allows  a  quiek  understanding  of  the  performanee  and  trends. 

For  eaeh  seleeted  performanee  parameter,  a  planned  performanee  profile  must  be  established. 
This  profile  establishes  the  performanee  expeeted  over  the  life  of  the  projeet.  Planned  profiles 
may  refleet  eonstant  values.  This  would  probably  be  assoeiated  with  teehnieally  mature,  low-risk 
eontraet  items.  In  this  ease,  the  profile  would  appear  as  a  horizontal  line  against  time.  The 
requirement  is  the  same  at  eaeh  major  milestone.  However,  most  development  items  are  not 
expeeted  to  refleet  mature  values  during  initial  analysis  and  testing,  sinee  the  TPMs  are  generally 
ehosen  to  provide  insight  into  areas  of  higher  performanee  risk.  The  profiles  for  these  items  will 
be  a  diagonal  or  eurved  line  that  represents  the  expeeted  performanee  growth  over  time.  The 
growth  is  expeeted  due  to  improvements  planned  in  hardware,  software,  and  operational 
proeedures  that  are  refleeted  by  the  TPM.  The  value  of  the  point  at  the  end  of  development  is 
based  on  the  speeified  requirement  of  the  TPM  parameter.  It  is  advisable  to  inelude  any  major 
milestones  or  design/performanee  events  on  the  profile. 

The  profile  and  the  aetual  measured  TPM  values  are  plotted  and  analyzed  together.  Over  time, 
the  measured  values,  predieted  values  and  parameter  requirement  should  eonverge.  Planned 
profiles  should  not  be  viewed  as  statie,  partieularly  where  Systems  Engineering/engineering 
development  is  still  in  proeess.  Where  trade  studies  indieate  that  eost,  or  time,  to  aehieve  a 
planned  requirement  is  exeessive,  the  requirement  values  should  be  re-evaluated. 

The  thresholds  bound  the  range  of  aeeeptable  performanee  for  the  attribute  of  the  system  element 
being  measured  in  its  delivered  state.  The  thresholds  are  established  based  on  faetors  sueh  as 
teehnieal  requirements,  toleranees,  and  eost-performanee  trades.  Toleranee  bands  are  defined  to 
refleet  aeeeptable  risk  limits  assoeiated  with  aehieving  the  performanee  measured  by  the  TPM 
based  on  the  planned  performanee  profile.  They  are  plaeed  on  eaeh  side  of  the  planned 
performanee  profile,  representing  estimating  error,  and  define  the  region  in  whieh  it  is  reasonable 
to  expeet  that  the  thresholds  will  be  attained  within  projeet  eonstraints.  Two-sided  toleranee 
bands  are  not  always  possible  or  neeessary,  depending  on  the  situation. 

The  toleranee  bands  ean  be  subdivided  into  risk  bands,  whieh  are  regions  of  low,  moderate,  and 
high  risks.  For  example,  risk  bands  may  be  defined  sueh  that  a  0  to  -5%  deviation  from  the 
required  value  is  eonsidered  Green  (low  risk),  -5  to  -10%  is  eonsidered  Yellow  (moderate  risk), 
and  less  than  -10%  is  eonsidered  Red  (high  risk).  Different  bands  may  be  appropriate  for  eaeh 
TPM  and  the  values  of  the  bands  ehange  with  the  planned  performanee  profile.  Establishment  of 
toleranee  and  risk  bands  depends  on  several  faetors  sueh  as  life  eyele  stage,  defined  thresholds, 
priority  of  the  performanee  attribute,  type  of  requirement,  teehnology  applied,  and  teehnieal 
preeedenee.  Risk  bands  ean  inelude  TPM  values  that  exeeed  the  defined  performanee 
requirement,  if  it  affeets  the  eost-effeetiveness  of  the  solution.  Assoeiations  between  teehnieal 
risk  assessed  by  the  TPMs  and  the  projeet  risk  assessed  by  eost  and  sehedule  measures  should  be 
understood  and  defined. 


27  Deeember  2005 


33 


Technical  Measurement 


INCOSE-TP-2003-020-01 


TPM  parameters  are  expeeted  to  be  added,  dropped,  or  modified  as  the  life  cycle  progresses. 
Since  TPMs  (and  MOPs)  provide  insight  into  technical  risks,  it  is  expected  that  the  values  will 
change  as  the  risks  of  the  project  or  their  relative  priorities  change.  TPMs  will  be  added  when 
new  risks  are  identified  or  existing  risks  are  raised  in  priority.  This  may  be  the  result  of  changes 
in  requirements/needs,  risks  being  mitigated,  or  risks  that  were  initially  overlooked.  TPMs  may 
not  need  to  be  monitored  when  their  value  improves  beyond  the  requirement,  reducing  the  risk  to 
acceptable  levels,  or  if  there  is  no  further  basis  for  expectations  of  future  changes  in  their  value. 
Modifications  to  TPMs  occur  when  the  parameter  values  of  the  associated  requirements  change 
or  changes  in  the  measurement/  analysis  techniques  are  deemed  necessary  to  provide  adequate 
insight.  For  example,  the  tolerance  bands  may  also  change  when  trades  are  made  between  the 
technical  performance  parameters.  If  some  of  the  margin  for  a  performance  parameter  is  traded 
to  reduce  risk  in  another  parameter,  then  the  tolerance  bands  for  the  TPMs  associated  with  these 
parameters  will  change  accordingly.  Modifications  should  be  recorded  with  rationale  and 
approvals  and  maintained  per  organizational  policy. 

Thus,  when  defining  the  technical  measures,  in  addition  to  the  information  suggested  in  the 
definition  of  measures  by  PSM  or  INCOSE,  it  is  good  to  include  the  following  items  of 
information: 

•  Requirement,  goal,  or  KPP  the  measure  is  traceable  to 

•  Specific  performance  risk  or  issue  addressed 

•  Thresholds,  tolerance  limits,  and  risk  bands  that  apply 

•  Definition  of  the  performance  profile  expected 

•  Related  WBS  element 

•  Applicable  part(s)  of  the  life  cycle 

6.3.5  Technical  Measurement  Relationship  To  The  WBS 

To  serve  as  a  more  comprehensive  management  tool,  the  TPM  specification  may  identify  the 
Work  Breakdown  Structure  (WBS)  element  associated  with  it.  This  association  provides  a 
connection  to  the  Earned  Value  Management  System  (EVMS)  and  to  the  schedule.  This 
facilitates  management  understanding  of  the  cost,  schedule  and  technical  relationships.  It  also 
provides  a  complete  picture  of  product  status,  detection  and  understanding  of  development 
problems,  and  identification  of  emerging  project  and  technical  risks.  When  technical  risks  or 
problems  are  identified,  the  cost  and  schedule  impacts  can  be  determined  in  a  more  expeditious 
manner.  However,  this  is  not  always  practical  or  cost-effective,  if  the  technical  measures  are  not 
well  correlated  to  the  WBS  and  system  architecture  elements.  Also,  a  specific  technical  measure 
may  not  be  associated  with  all  of  the  lower  level  WBS  elements. 

A  parameter  tree,  which  defines  the  hierarchy  and  interrelationships  of  the  technical  measures, 
can  be  developed  from  the  product  elements  of  the  WBS.  Developing  the  TPM  parameter  tree 
from  the  WBS  ensures  traceability  of  progress  on  technical  performance  to  cost  and  schedule 
aspects  of  the  work  effort  (through  the  cost/schedule  control  system).  Project  management  can 
then  associate  technical  performance  variances  (such  as  a  weight  parameter  exceeding  the 
tolerance  limit  by  10  percent)  with  schedule  and  budgetary  status  (e.g.,  80  percent  of  budget 
expended,  less  than  one  month  for  final  value  to  be  achieved).  Note  that  the  TPM  parameter  tree 
may  not  perfectly  correspond  to  the  WBS  in  content  or  degree  of  detail,  for  example,  in  solving 
a  complex  software  problem,  it  may  be  necessary  to  expand  certain  parts  of  the  WBS  to  facilitate 
parameter  tracking  in  that  area,  figure  6-3  below  shows  an  example  of  a  parameter  tree  that 
illustrates  the  TPM  to  WBS  relationship. 
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Figure  6-3  Example  of  Tracking  TPMs  to  WBS  Elements'' 


6.3.6  Summary  of  Selection  Guidance 

Table  6-4  contains  guidance  that  has  been  derived  to  assist  in  the  MOE,  MOP,  and  TPM 
selection. 


DSMC  Teaching  Note,  “Technical  Performance  Measurement”,  Robert  Lightsey,  October  1997 
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MOE 

MOP 

TPM 

Provides  insight  into  at  least  one 
operational  objective  or  mission 
requirement 

Should  enable  calculation  of  or 
insight  into  at  least  one  MOE 

Technical  risks  (including 
feasibility  of  requirements) 

MOEs  should  not  be  strongly 
correlated:  provide  insight  into 
different  aspects  of  the  operational 
alternative 

A  subset  should  be  based  on 
KPPs 

Should  have  ability  to  support 
trades  among  possible  solutions 
for  achieving  Key  Performance 
Parameters  (KPP) 

Select  and  define  in  the  context  of 
the  operational  objective  :  no 
predefined  MOEs/values 

May  be  related  for  insight  into  a 
specific  system  characteristic  or 
alternative 

Strongly  influenced  by  quality 
requirements 

Select  and  define  independent  of 
the  alternatives  at  hand  :  represent 
an  independent  means  to 
collectively  evaluate  the 
alternatives 

Focus  on  technical  risks  and 
support  trades  of  alternative 
solutions  at  the  system  level 

Should  have  ability  to  track 
adherence  to  technical 
constraints  (including  physical, 
technology,  and  performance) 

Select  only  a  few  MOE/MOPs  : 
may  be  an  order  of  magnitude 
more  TPMs 

Should  collectively  provide 
insight  into  system  affordability 

Should  have  ability  to  show 
relationship  to  risks,  cost, 
schedule  &  quality  objectives 

Each  KPP  should  have  an 
associated  MOE  or  MOP 

Should  be  able  to  be  linked  to 
future  testing  of  alternatives 
and  chosen  KPPs 

May  be  traceable  to  applicable 
WBS  elements 

Table  6-4  MOE/MOP/TPM  Selection  Guidance 

TPMs  are  expected  to  be  traceable  to  the  needs  of  the  user,  while  providing  concrete  technical 
measurement  that  can  be  projected  and  tracked  as  the  project  progresses.  For  example,  a  user 
may  have  an  operational  requirement  for  the  technical  solution  to  be  survivable  under  combat 
conditions  for  a  specified  number  of  missions.  The  survivability  as  specified  may  represent  an 
MOE,  and  if  considered  critical  enough,  a  KPP.  Direct  determination  of  the  achievement  of  the 
survivability  requirement  is  dependent  on  system  level  test  and  demonstration  under  the  intended 
operating  conditions.  This  cannot  be  determined  by  itself  until  the  technical  solution  is  fully 
developed.  However,  there  are  measurable  parameters  of  the  technical  solution  that  are  relevant 
factors  for  survivability,  such  as  the  probability  of  escaping  a  hit.  This  could  be  considered  the 
MOP,  a  measure  of  an  attribute  of  the  technical  solution  at  the  system  level.  This  MOP  can  then 
be  assessed  by  looking  at  lower  level  factors  that  can  be  measured  periodically  as  the  technical 
solution  is  developed,  such  as  radar  accuracy,  velocity,  target  accuracy,  kill  efficiency,  and  firing 
rate.  These  are  some  of  the  potential  TPMs.  Even  though  these  TPMs  may  be  directly 
measurable,  they  may  have  lower  level  technical  parameters  on  which  they  are  dependent  that 
are  easily  measurable  at  more  frequent  periods  or  earlier  in  the  development,  such  as  velocity  is 
dependent  on  weight,  dimensions,  etc.  These  lower  level  TPMs  can  provide  early  feedback  for 
high-risk  areas  of  the  design. 

Therefore,  the  technical  manager  can  select  and  track  the  factors  that  are  most  applicable  to  the 
technical  solution  and  level  of  risk.  The  decision  on  selection  of  parameters  for  TPM  tracking 
must  also  take  into  consideration  the  extent  to  which  the  parameter  behavior  can  be  projected 
(profiled  over  a  time  period),  whether  data  will  be  available,  and  whether  or  not  it  can  actually  be 
measured.  If  the  parameter  cannot  be  profiled,  measured,  or  is  not  critical  to  project  success,  then 
it  should  not  be  selected  as  a  TPM. 
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6.4  Integrate  Into  The  Project  Processes 

It  is  important  to  understand  and  plan  for  the  interaetion  of  the  measurement  activities  with  both 
the  technical  and  management  processes  of  the  project.  Figure  6-5  illustrates  the  key  interactions 
of  these  processes  with  the  measurement  activities.  The  technical  processes  will  aid  planning 
and  provide  essential  insight  into  risk/opportunity  and  quality  management  of  the  technical 
solution  through  the  life  cycle.  The  management  processes  require  information  to  enable  good 
and  timely  decisions.  The  information  needs  of  these  management  processes  are  considered 
together  with  the  technical  requirements  and  KPPs  of  the  technical  processes  to  form  the  inputs 
to  the  selection  and  definition  of  measures.  The  definitions  of  the  measures  determine  the  data 
collection  requirements  (also  known  as  base  measures).  As  the  data  is  collected,  it  needs  to  be 
processed,  analyzed  and  made  accessible  to  management  for  review  and  to  determine  whether 
actions  are  warranted.  This  integration  needs  to  be  defined  in  the  plan,  including  timing, 
milestones,  responsibilities,  etc.  The  project  team  needs  to  ensure  that  the  data  to  be  collected  is 
cost-effective  to  collect  versus  the  insight  provided  and  can  be  collected  as  the  process  is 
executed.  The  determination  of  the  fit  of  data  collection  to  the  processes  should  consider  the 
measurement  specifications,  the  acquirer  and  supplier  processes,  and  decision  points. 
Responsibility  for  tracking  the  technical  measures  throughout  the  project  must  be  assigned. 
Typically,  this  is  assigned  to  an  organization,  IPT,  or  individual  with  overall  system  oversight. 


Note:  Not  all  relationships  shown  -  only  those  relevant  to  this  discussion. 


Figure  6-5  Integration  of  Technical  Measurement  with  Project  Processes 

Failure  to  achieve  the  technical  measures  (especially  TPMs)  often  results  in  significant 
cost/schedule  impacts  to  projects,  and  may  result  in  penalties  or  project  cancellation.  Where  the 
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measure  indieates  margin  exists  for  the  teehnical  requirement  being  tracked,  there  may  be  an 
opportunity  to  use  the  margin  in  a  technical  trade  to  reduce  the  overall  system  risk.  Therefore, 
these  measures  are  always  candidates  for  incorporation  into  the  project’s  Risk  and  Opportunity 
Management  program.  Interaction  with  the  risk  and  opportunity  management  process  includes 
providing  the  list  of  TPM  parameters  for  risk  assessment;  providing  the  TPM  status/profile 
charts  for  risk  analysis;  and  receiving  the  risk  "watch  list"  from  the  risk  and  opportunity 
management  process,  for  potential  additional  TPM  parameters. 
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7  Performing  Technical  Measurement 

The  overall  proeess  activities  for  performing  the  technical  measurement  are  the  same  as  those 
defined  in  the  PSM  and  INCOSE  documents.  This  includes  the  following  activities,  which  will 
be  discussed  in  this  section  to  the  extent  that  supplemental  information  or  activity  variations 
apply. 

•  Collect  and  process  data 

•  Analyze  data  (from  technical  measures) 

•  Make  recommendations 

7. 1  Collect  And  Process  Data 

The  data  collected  will  vary  with  the  point  in  the  life  cycle  and  the  test  environment.  Early  in 
the  life  cycle  the  data  is  likely  to  be  based  on  modeling,  simulation  and  analysis.  As  the  life 
cycle  continues  and  implementation  begins,  the  data  will  be  obtained  from  the  test  environments, 
first  at  the  lower  levels  of  the  product  realization,  then  from  the  various  levels  of  integration 
testing.  During  early  stages  of  testing,  the  test  environment  may  not  be  representative  of  the 
intended  operational  environment.  In  this  situation,  the  direct  measurement  of  the  tests  should  be 
supplemented  with  data  from  similar  systems  and  estimates  to  produce  the  most  meaningful 
results.  During  qualification  testing,  the  test  environment  is  usually  very  representative  of  the 
intended  operating  environment.  Therefore,  the  test  data  may  be  used  directly  to  calculate  the 
measures.  At  all  points  in  the  life  cycle,  data  should  be  collected  at  a  level  appropriate  to  provide 
insight  into  KPP  progress  and  to  localize  issues. 

Regarding  the  aggregation  and  normalization  of  data,  this  data  processing  should  account  for  the 
relationships  of  the  TPMs  to  the  MOPs  and  the  MOPs  to  the  MOEs.  As  discussed  previously, 
the  TPMs,  MOPs,  and  MOEs  often  have  a  hierarchical  relationship.  The  TPMs  will  be 
aggregated  or  processed  using  defined  functions  to  provide  MOPs  or  insight  into  the  MOPs. 
Eikewise,  the  MOPs  are  aggregated  or  are  used  as  input  to  functions  to  provide  MOEs  or  insight 
into  the  MOEs.  The  aggregation  functions  or  methods  should  be  defined  in  the  measurement 
specification.  Sample  methods  for  the  aggregation  include: 

1 .  Known  algebraic  relationships 

2.  Decision  analysis  methods,  such  as  pair-wise  comparisons,  trend  analysis,  objective 
matrix  analysis,  and  operations  research  analysis 

3.  Expert  judgment  with  or  without  use  of  statistical  techniques,  such  as  Delphi,  surveys  or 
questionnaires,  and  nonparametric  methods 

It  is  good  practice  to  establish  a  repository  for  technical  data,  documentation,  analysis,  and 
reports  to  facilitate  access  by  the  various  stakeholders.  Access  should  be  provided  according  to 
the  measurement  plan  based  on  the  role  and  information  needs  of  the  stakeholders. 

7.2  Analyze  Data 

7.2.1  General  Technical  Measurement  Analysis 

Per  the  PSM  guidebook,  the  analysis  task  is  performed  in  order  to  convert  data  into  information 
that  can  be  used  by  managers  to  support  decision-making.  Typical  methods  include  engineering 
analysis,  modeling  and  simulation  (including  scale  models,  and  mathematical  models). 


27  December  2005 


39 


Technical  Measurement 


INCOSE-TP-2003-020-01 


comparison,  and  various  levels  of  system  element  test.  The  methods  or  tools  chosen  for  the 
analysis  is  related  to  such  factors  as: 

•  Type  of  parameters  selected 

•  Maturity  of  project 

•  Data  availability 

•  Information  needs  of  the  project 

•  Resource  availability  (including  test,  analysis,  modeling,  or  simulation  assets) 

•  Cost  of  the  method  versus  insight  desired 

•  Life  cyele  stage 

•  Type  of  performance  being  assessed 

•  Required  fidelity 

•  Associated  risk  levels  reflected  by  the  measure. 

Eaeh  method  has  assoeiated  eosts,  response  times  and  fidelity,  thus  providing  different  levels  of 
confidenee.  The  confidence  level  needed  should  be  weighed  against  the  resources  required. 
Choose  the  lowest  cost  method  that  will  provide  the  eonfidence  level  needed  at  that  point  in  the 
project  and  is  commensurate  with  the  risk. 

At  the  completion  of  each  evaluation,  analysis,  or  test,  results  should  be  reeorded  for  comparison 
with  planned  values;  any  variances  must  be  analyzed.  The  analysis  ineludes  evaluation  of  the 
effect  of  variances  on  the  teehnical  projeet  risk,  sehedule,  and  cost.  Summary  performance 
status  reports  should  be  prepared  from  the  basie  parameter  status  data  provided  by  the  technieal 
measurement. 

“The  performing  activity  determines  the  achievement-to-date  for  eaeh  teehnieal  parameter. 
Teehnical  progress  is  assessed  in  terms  of  both  allowed  variation  and  the  trend  in  achievement- 
to-date  compared  with  the  planned  value  profile.  When  progress  in  the  technieal  effort  supports 
revision  of  the  current  estimate,  a  new  profile  and  current  estimate  is  developed.  Risk 
assessments  and  analyses  are  updated  to  reflect  changes  in  planned  value  profiles  and  current 
estimates,  and  impacts  on  related  parameters. 

For  identified  deficiencies,  analyses  are  performed  to  determine  the  cause(s)  and  to  assess  the 
impacts  on  higher-level  parameters,  interfaces,  and  system  eost  effeetiveness.  Alternative 
recovery  plans  are  developed  with  cost,  schedule,  performance,  and  risk  impacts  fully  explored. 
For  performance  in  excess  of  requirements,  the  marginal  cost  benefits  and  opportunities  for 
realloeation  of  requirements  and  resourees  are  assessed  and  an  appropriate  eourse  of  aetion 
defined.”^ 

Since  eomplex  system  evaluation  may  involve  the  assessment  of  several  major  system  elements 
through  somewhat  independent  MOFs,  the  total  system  evaluation  needs  to  integrate  each  of  the 
system  element  assessments.  The  method  for  integrating  the  assessments  should  be  defined 
when  the  MOFs  are  selected  and  requires  knowledge  of  how  the  elements  interaet. 

In  addition  to  providing  technieal  risk  insight,  the  predietive  trend  analysis  of  the  TPMs  also 
serves  as  a  leading  indicator  for  risk  insight  from  the  project  perspective.  This  is  beeause 


^  Systems  Engineering  Guide,  Version  1.1,  Seetion  2. 3. 4. 8,  Air  Foree,  April  1996 
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problems  with  the  teehnieal  progress  often  show  up  before  the  projeet  impaet  is  seen  as  a 
traditional  earned  value  eontrol  aceount  varianee. 

During  system  design  and  development,  “Achieved-to-Date”  values  should  be  traeked  at  each 
assessment  milestone,  for  each  selected  parameter,  and  at  each  specified  level  of  the  WBS.  These 
point  estimates,  based  on  either  analysis  or  testing,  are  then  used  to  forecast  the  expected  value 
that  will  be  achieved. 

Due  to  the  interrelationships  of  cost,  schedule,  and  technical  attributes  of  providing  a  technical 
solution,  it  is  important  to  assess  the  technical  measures  together  with  cost  and  schedule 
measures.  (See  MIL  STD  499B,  section  6.4.8  or  EIA  632/IS,  section  5.2.9) 

7.2.2  Rigor  Of  Analysis  Techniques 

Analytical  techniques  can  be  classified  as  quantitative  or  qualitative.  Quantitative  analytical 
techniques  are  objective  and  based  on  the  ability  to  describe  issues  in  terms  of  mathematical 
relationships  that  allow  the  use  of  computational  methods,  modeling,  and  simulation.  Qualitative 
techniques  can  combine  objective  and  subjective  input  and  relies  on  judgments  based  on 
experience  (usually  of  ‘experts’  such  as  using  Delphi  Analysis)  and  analogies/comparisons.  The 
list  below  describes  some  advantages  and  disadvantages  of  both  quantitative  and  qualitative 
analysis. 

•  Quantitative  Analysis 

o  Advantages 

■  Repeatable 

■  Supports  parametrie  analysis 

■  Reduees  bias;  Makes  existing  biases  more  visible 

■  Usually  higher  fidelity  or  aeeuraey,  if  valid  data  used 
o  Disadvantages 

■  Requires  signifieant  input  data 

■  Requires  signifieant  time  and  skill  to  produee  and  interpret  answers 

■  Requires  understanding  mathematieal  relationships 

■  Results  may  be  diffieult  to  understand  by  those  not  performing  the  analysis  if  the  analysis 
is  fairly  eomplex 

•  Qualitative  Analysis 

o  Advantages 

■  Provides  quiek  answers 

■  Requires  little  quantitative  input  data 

■  Doesn’t  require  understanding  mathematieal  relationships 

■  Applieable  to  eomplex  subjeetive  issues 

■  Requires  roughly  the  same  effort  regardless  of  issue  eomplexity 
o  Disadvantages 

■  Influeneed  by  seleetion  of  experts  (no  guarantee  of  repeatability) 

■  Not  well-suited  to  parametrie  analysis 

■  Variable  expertise 

■  Experts  with  narrow  or  widely-divergent  interests 

■  May  not  use  best  qualified  experts  (not  identified,  not  available,  ean’t  afford) 

■  Results  diffieult  to  interpret  relative  to  quantitative  goals 

The  advantages  of  one  technique  are  often  the  disadvantages  of  the  other.  In  general,  quantitative 
techniques,  when  practical,  are  significantly  preferable  to  qualitative  techniques.  Qualitative 
techniques  include  Delphi  Analysis,  Analytical  Hierarchy  Process  (AHP),  and  Value  Focused 
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Thinking.  The  qualitative  teehniques,  are  quieker  and  eost  less,  but  provide  lower  eonfidenee 
levels. 

7.2.3  Estimation  Analysis 

As  presented  in  the  PSM  guidebook  (Part  4),  the  analysis  task  ineludes  estimation  analysis, 
feasibility  analysis,  and  performanee  analysis.  Estimation  is  essential  to  project/produet 
planning,  first  for  initial  planning,  then  for  subsequent  re-planning  as  some  data  beeomes 
available  and  trends  ean  be  determined.  Due  to  this,  estimation  is  often  the  first  type  of  analysis 
eneountered  on  a  projeet.  With  respeet  to  the  teehnieal  measurement,  estimation  is  eondueted  to 
provide  projeetions  of  product  attributes,  quality,  and  performance,  including  the  establishment 
of  initial  performance  profiles  with  tolerance  bands  and  thresholds.  Estimation  is  conducted  to 
establish  target  values  or  numerical  expectations  for  subsequent  activities  and  parameters,  based 
on  currently  available  data.  “Estimation  uses  one  measure  to  predict  the  value  of  another.”  Eor 
example,  many  technical  parameters  have  direct  relationships,  such  as  weight  and  velocity  or 
acceleration.  The  first  column  of  Table  7-1  contains  some  summary  guidance  based  on  lessons 
learned  for  performing  estimation  analysis. 

It  is  necessary  to  re-evaluate  the  estimates  at  key  milestones  in  the  life  cycle,  especially  as  more 
actual  data  becomes  available.  The  new  estimates  will  have  higher  fidelity,  since  they  are  based, 
to  some  extent,  on  actual  results.  As  estimates  are  found  to  be  inadequate  or  incorrect,  it  is 
important  to  understand  other  factors  that  may  be  affected  by  the  estimate  and  account  for  those 
relationships  in  any  re-planning.  This  includes  understanding  that  changes  in  technical 
parameter  estimates  may  have  impacts  on  cost,  schedule,  other  resources,  processes,  etc.  The 
PSM  guidebook  has  an  analysis  model  that  describes  these  relationships  in  a  generic  sense.  The 
need  to  change  point  estimates  will  generally  result  in  changes  to  other  parts  of  the  performance 
profde.  Table-7-1  also  includes  specific  guidance  for  updating  estimates.  Additional  general 
guidance  for  estimation  can  be  found  in  the  PSM  guidebook  (Part  4). 

7.2.4  Feasibility  Analysis 

Eeasibility  analysis  assesses  the  likelihood  of  achieving  the  estimated  values  per  the  progress 
plan  or  performance  profile.  This  analysis  is  conducted  during  the  initial  planning  and  at  all 
subsequent  re-plans  (due  to  changes  in  functionality,  availability  of  actual  project  data,  changes 
in  assumptions,  or  failure  to  meet  current  plans).  The  feasibility  analysis  either  confirms  the 
planning  that  resulted  from  the  estimation  analysis  or  provides  alternatives  that  have  a  higher 
probability  of  achievement.  Since  only  parts  of  the  plan  cause  the  plan  to  be  infeasible,  it  is 
important  to  quickly  understand  and  correct  those  parts.  To  reach  acceptable  alternatives,  it  may 
be  necessary  to  consider  trades  among  the  product,  project,  and  process  attributes. 

The  feasibility  analysis  is  intended  to  look  at  the  following: 

•  Basis  of  the  estimate 

•  Realism  of  adjustments 

•  Confidence  in  the  estimation  and  estimation  techniques 

•  Validity  of  or  changes  in  assumptions 

•  Changes  in  project/product  attributes  that  may  affect  the  estimate 

•  Comparisons  of  related  KPPs  or  other  relevant  parameters 

The  following  are  a  few  key  characteristics  of  a  feasible  plan  for  technical  achievement: 
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•  The  target  values  support  aehievement  of  overall  projeet  and  produet  objeetives 

•  The  rate  of  planned  progress  is  reasonable  (aehievable  based  on  historical  performance 
and  current  technology) 

•  Planned  performance  is  consistent  with  past  performance  capability  as  adjusted  for  new 
processes  and  technology  (rationale  should  be  understood  and  agreed  to  for  process  and 
technology  adjustments) 

•  Targets  values,  such  as  operational  availability,  should  be  reasonable  within  the  context 
of  use 

•  Risks,  especially  for  unprecedented  target  values,  have  been  identified  and  accounted  for 

Table  7-1  contains  some  guidance  based  on  lessons  learned  for  performing  feasibility  analysis. 
Additional  general  guidance  for  feasibility  analysis  can  be  found  in  the  PSM  guidebook  (Part  4). 

7.2.5  Performance  Analysis 

Performance  analysis  is  conducted  to  determine  whether  the  development  of  the  technical 
solution  is  meeting  the  plans,  assumptions  and  targets.  If  a  performance  profile  has  been 
established,  the  performance  analysis  examines  whether  the  TPM  is  within  the  established 
tolerance  bands  and  is  likely  to  achieve  the  required  thresholds.  Performance  analysis  should  be 
conducted  periodically  after  the  project  has  committed  to  a  technical  plan. 

Performance  deviations  that  exceed  the  tolerance  bands  of  the  planned  parameter  values  of  the 
performance  profile  reflect  problems  that  must  be  addressed.  Evidence  of  trends  that  are  likely 
to  exceed  the  tolerance  bands  in  the  future,  if  not  addressed,  reflect  risks  that  may  require 
mitigations.  Identification  and  analysis  of  trends  is  essential  to  provide  predictive  insight  and 
allow  management  to  take  action  before  the  associated  risks  turn  into  problems.  When  these 
risks  or  problems  are  identified,  it  is  necessary  to  determine  their  cause  and  assess  the  potential 
impact  on  higher-level  parameters  (higher-level  TPMs,  MOPs,  and  MOEs),  interface 
requirements,  and  the  overall  value  and  cost-effectiveness  of  the  technical  solution.  Where 
trends  exist  and  tolerance  bands  have  not  yet  been  exceeded,  performance  outcomes  should  be 
predicted  by  extrapolating  current  trends  and  considering  future  events  and  influences.  Eor  these 
problems  and  higher-rated  risks  (i.e.,  likely  to  exceed  tolerance  bands),  alternatives  courses  of 
actions  should  be  developed  showing  the  expected  cost,  schedule  and  technical  impact  of  each 
alternative  and  the  evaluation  of  the  alternatives  against  established  criteria.  Where  performance 
exceeds  the  tolerance  bands,  such  that  there  is  an  overachievement  of  the  requirement, 
opportunities  for  reallocation  of  requirements  and  resources  should  be  examined  and  tracked. 

When  the  performance  has  been  successful,  it  is  good  practice  to  understand  the  success  in  terms 
of  the  planning,  estimation,  feasibility  analysis,  and  corrective  or  preventive  actions  put  in  place. 
This  is  the  basis  of  valuable  lessons  learned  for  incorporation  into  the  standard  processes  and  for 
future  project  work. 

It  may  be  necessary  to  generate  additional  indicators  to  provide  the  insight  to  complete  this  task. 
Two  types  of  indicators  that  are  highly  applicable  for  technical  measurement  are  trend-based 
indicators  and  threshold/target-based  indicators.  Table-7-1  contains  some  guidance  based  on 
lessons  learned  for  executing  performance  analysis.  Additional  general  guidance  for 
performance  analysis  can  be  found  in  the  PSM  guidebook  (Part  4)  and  the  PSM  book,  chapters  4 
and  5. 
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7.2.6  Summary  Of  Analysis  Guidance  For  Technical  Measurement 

Table  7-1  provides  a  tabular  summary  of  the  analysis  guidanee  for  MOEs,  MOPs  and  TPMs  with 
respect  to  estimation,  feasibility,  and  performance  analysis  tasks.  The  sequence  of  these  tasks  is 
dependent  of  project-specific  factors. 


Estimation  Analysis 

Feasibility  Analysis 

Performance  Analysis 

Identify  predictions  needed,  key 
drivers  of  variation,  relationships 
between  measurable  attributes  of 
the  drivers 

Determine  risk,  cost, 
schedule,  and  quality 
impacts  reflected  by  the  KPP 
estimated  values 

Track  actual  performance 
against  progress  profiles/plans 

Collect  data  for  the  attributes  of 
the  drivers 

Investigate  whether: 

•  Performance  parameters 
have  been  achieved 
before 

•  Current  technology 
supports  the  desired 
performance  within  known 
constraints 

•  Relationships  of  identified 
risks 

•  Degree  of  control  over  the 
risks  to  progress 

Show  acceptable  ranges  of 
variation  that  correspond  with 
risks  and  constraints 

Determine  quantitative 
relationships/models 
•  May  include  data  from  other 
indicators 

Identify  variances  of  achieved 
values  from  plan 
•  Variances  indicate  the 
current  level  of  risk,  which 
may  result  from  process, 
technology,  or  product 
attributes 

Generate  estimate  using  model 

Adjust  estimate  as  necessary  to 
account  for: 

•  Engineering  trades 

•  Technology  capability  and 
constraints 

Assess  impacts  of  the 
variances 

Establish  expected  growth 
profiles  or  other  progress  plans 
for  the  indicators  based  on 
estimates 

If  risk/impact  is 
unacceptable,  then: 

•  Identify  alternative  design 
solutions  and  estimates, 
or 

•  Identify  and  implement 
risk  handling  strategies 

Periodically  assess  achieved 
values 

•  Understand  success  of 
corrective/preventive  actions 

•  Identify  new  risks 

Track  the  indicator  against  the 
progress  plan/profile  and  update 
estimates  with  partial  actual  data 
as  it  becomes  available.  Apply 
rolling  wave  planning  techniques, 
where  applicable. 

Establish  confidence  intervals 
based  on  amount  of  actual  data 

Perform  decision  analysis 
and  take  action 

Table  7-1:  MOE/MOP/TPM  Analysis  Guidance 


7.2.7  Tracking  And  Providing  Status  Of  Technical  Measures 

Table  7-2  provides  an  example  of  a  tracking  and  status  form  for  technical  measurement.  It 
includes  the  parameter  of  interest,  the  required  value,  its  relationship  in  the  design  architecture, 
the  predicted  performance  based  on  estimation  or  current  trends,  and  difference  between  the 
predicted  value  and  the  requirement,  and  the  means  by  which  the  prediction  was  achieved.  Each 
project  should  define  a  way  to  capture  and  track  this  information. 


27  December  2005 


44 


Technical  Measurement 


INCOSE-TP-2003-020-01 


Parameter 

Title 

Goal  or 
Requirement 

Next  Level  of 
Design 
Concurrence 

Predicted 

Performance/ 

Current 

Value 

Difference 

Justification 

For 

Prediction 
(i.e.,  Analysis, 
Test  Results, 
Similarity, 
etc.) 

TPM  Status 

Critical 

Availability 

Requirement  = 
99.8% 

Goal  =  99.88% 

Attitude  Control 
Subsystem 

99.96% 

+0.16% 

Analysis  using 
limited/early 
test  data 

Green 

Phase  Noise- 
100  Msps 

Requirement  = 

1.2  RMS 

Goal  =  0.90  RMS 

Signal  Data 
Distributor 

0.75  RMS 

0.45  RMS  better 
than  required 
value 

Test  Results 

Green 

Signal  Latency 

Requirement  = 

=/-  30  ns 

Goal  =  +/-  27ns 

Signal  Data 
Distributor 

+/-  32.8  ns 

2.8  ns  beyond 
required  value 

Test  Results 

Red 

Table  7-2  Example  Technical  Measurement  Tracking  and  Status  Form 

In  addition  to  the  composite  tabular  tracking,  it  is  often  desirable  to  have  individual  reports  with 
graphics  and  narratives  for  the  measures.  The  graphic  will  usually  allow  the  analyst  and 
management  to  more  easily  identify  trends  that  could  be  addressed  before  they  turn  into 
problems.  “The  graphic  [see  example  in  Figure  3-1]  shows  the  projected  behavior  of  the  selected 
parameter  as  a  function  of  time,  and  further  shows  actual  observations,  so  that  trends  and 
deviations  from  plan  can  be  identified  and  assessed.  The  narrative  portion  of  the  report  should 
explain  the  graphic,  addressing  the  reasons  for  trends  or  deviations  from  plan,  assessing  the 
potential  impact  of  the  trend  or  seriousness  of  those  deviations,  explaining  actions  underway  to 
correct  the  situation  if  required,  and  projecting  future  performance,  given  the  current  situation.”^ 


7.3  Make  Recommendations 
7.3.1  General  Guidance 

It  is  important  to  report  analysis  results  and  recommendations  to  management  in  a  timely, 
efficient  and  effective  manner.  The  communication  of  results  should  include; 

•  Overall  evaluation  of  the  technical  status,  including  known  issues  and  projections  of 
performance  through  completion 

•  Identification  and  assessment  of  any  specific  risks,  problems,  and  lack  of  information. 
This  should  include  identification  of  specific  outliers  or  trends,  a  discussion  of  their 
location,  causes,  and  impacts 

•  Recommendations  based  on  evaluation  of  alternative  actions.  Include  a  brief  discussion 
of  the  alternatives  and  their  evaluation  against  established  criteria.  Highlight  the 
advantages  and  disadvantages  of  each 

•  Potential  new  risks  to  be  aware  of,  including  those  that  may  result  from  implementation 
of  proposed  actions 


^  Adapted  from  DSMC  Teaching  Note,  “Technical  Performance  Measurement”,  Robert  Lightsey,  October  1997 
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•  An  indication  on  all  graphs  of  which  direction  is  considered  “good”,  making 
interpretation  easier 

Both  routine  reporting  and  ad  hoc  reporting  needs  to  be  provided.  Routine  reporting  includes 
monthly,  quarterly  and  annual  reports  essential  to  support  ongoing  management  knowledge  and 
decisions.  It  emphasizes  exceptions,  highlights  significant  new  information,  and  provides 
periodic  status.  The  frequeney  of  the  reporting  depends  in  part  on  the  life  cyele  stage.  Ad  hoc 
reporting  addresses  any  reporting  requirements  that  arise  from  various  stakeholders  on  a  request 
basis.  These  one-time  reports  provide  timely  information  to  stakeholders  for  unique  information 
needs  and  for  scheduled  reviews  or  milestones. 

Additional  general  guidance  for  making  decisions  from  measurement  results  can  be  found  in  the 
PSM  guidebook  (Part  4). 

7.3.2  Reporting  Analysis  Of  Alternative  Solutions 

During  the  analysis  of  alternative  solutions,  effectiveness  results  of  the  MOEs  need  to  be  clearly 
and  suceinctly  packaged,  and  the  manner  in  which  they  are  presented  should  minimize 
opportunities  to  mislead.  “The  basic  effectiveness  results  are  the  MOE  evaluations  for  each 
alternative.  Once  the  MOE  evaluations  have  been  presented,  it  may  also  make  sense  to  ‘roll  up’ 
these  results.  Rolling  up  results  describes  any  process  that  aggregates  results  for  individual 
alternatives.  A  roll  up  allows  comparing  the  alternatives  using  a  smaller  number  of  measures. 
The  advantage  of  having  a  smaller  number  of  measures  carries  the  obvious  disadvantage: 
information,  and  along  with  it  potential  insight,  is  lost  in  the  roll  up  process.  Aggregation  is 
acceptable  only  when  the  rationale  for  doing  it  is  sound.  This  means: 

•  The  aggregation  arises  naturally  from  relationships  among  the  MOEs 

•  The  significance  of  the  aggregates  is  clear 

•  The  aggregates  tell  a  clearer  story  than  the  individual  MOEs 

These  are  difficult  criteria  to  meet,  but  nothing  less  makes  good  sense.  The  message  is:  don't 
aggregate  just  to  aggregate.”^  finally,  weighting  of  MOEs  (different  values  (weights)  to 
different  MOEs),  should  only  be  done  with  input  from  the  decision  maker.  Although  it  is  rare 
that  the  MOEs  all  have  the  same  importance,  the  relative  weighting  needs  to  reflect  the  view  of 
the  decision  maker. 

7.3.3  Ongoing  Reporting  Though  The  Life  Cycle 

If  progress  toward  aehieving  performance  for  a  given  TPM  deviates  from  plan  beyond  the 
tolerance  bands  or  trends  exist  that  eould  exceed  the  tolerance  bands,  stakeholders  (including 
engineering  management,  projeet  management,  and  other  internal  &  external  customers)  should 
be  informed  so  impacts  can  be  minimized,  and  mitigation/contingency  plans  initiated.  The 
analysis  and  evaluation  of  alternative  actions  that  was  done  as  part  of  the  performanee  analysis 
should  be  summarized  and  presented  to  aid  the  decision-making.  Results  must  be  clearly 
understood  by  the  decision  makers.  Technieal  performance  (teehnical  risk)  should  be  addressed 
considering  its  effeet  on  project  performance  (cost  and  schedule  risk).  The  specific  corrective 
actions  or  mitigations  are  selected  based  on  the  risk  levels  assessed  and  their  impacts  on  other 
factors.  Before  acting  on  the  outcome  of  any  technieal  risk  assessment,  the  project  management 


^  Analysis  of  Alternatives  Handbook, ,  Seetion  6,  Office  of  Aerospace  Studies,  Air  Force  Materiel  Command, 
February  2002. 
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team  must  review  the  strengths  and  weaknesses  of  the  approach  and  understand  the  impacts  to 
technical,  cost,  and  schedule  aspects  of  the  project.  The  desired  actions  from  a  specific  technical 
perspective  may  not  be  possible  when  other  technical  and  project  factors  are  considered.  There 
may  be  a  need  to  perform  trades  within  the  project  and  technical  constraints.  The  resulting 
actions  may  include  replanning,  process  adjustments,  new  estimates,  changes  to  the  design 
solution,  incorporation  of  new  technologies,  reallocation  of  margins  (opportunities)  to  offset  risk 
areas,  etc.  In  some  cases,  the  deviation  or  trend  may  not  be  subject  to  immediate  action  due  to 
upcoming  events,  design  modifications,  tests,  analysis,  or  related  actions. 

The  performance  attributes  of  system  elements  that  reflect  technical  risk  in  a  project  should  be 
included  in  each  project  review.  Risk  handling  strategies  and  contingency  plans  with  trigger 
values  for  execution  should  be  established.  The  values  of  the  measures  should  be  reviewed 
against  these  “triggers”  to  determine  what  action,  if  any,  is  needed.  This  treatment  of  technical 
risk  is  an  essential  part  of  the  project  reviews. 

Table  7-3,  below,  is  a  simple  reporting  example  for  tracking  TPMs. 


System  Element 

TPM  Date  of  Profile: 

End  Product  Baseline 

ID# 

Name 

Units 

Parameter 

Achieved- 

To-Date 

Planned 

Value 

Demonstrated 

Technical 

Variance 

Current 

Estimate 

Threshold 

(System 

Requirement) 

Predicted 

Technical 

Variance 

Table  7-3  Example  Reporting  Table  for  Tracking  TPMs 
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8  Technical  Measurement  Checklist 

The  following  checklists  are  intended  to  assist  the  project  team  in  ensuring  the  necessary 
considerations  for  technical  measurement  have  been  accomplished.  They  are  applied  throughout 
the  life  cycle  by  the  project  personnel  assigned  to  perform  the  technical  measurement  tasks. 

-  General  - 

1.  Have  the  Technical  Measures  been  developed  with  a  common  understanding  of  the 
information  needs  from  the  relevant  stakeholders? 

2.  Are  the  Technical  Measures  for  this  project  identified?  Have  they  followed  the  selection 
and  specification  guidance  and  criteria? 

3.  Have  the  Technical  Measures  been  submitted  (and  incorporated  as  necessary)  into  the 
project  risk  and  opportunity  management  program? 

-  MOEs  - 

1 .  Are  the  MOEs  traceable  to  the  applicable  acquirer  needs,  goals,  objectives,  and  risks? 

2.  Are  the  MOEs  clearly  defined  with  associated  KPPs  identified? 

3.  Does  each  MOE  provide  insight  into  at  least  one  operational  objective  or  mission 
requirement? 

4.  Are  the  MOEs  independent  of  each  other;  each  MOE  provides  insight  into  different 
aspects  of  the  operational  alternative? 

5.  Are  the  MOEs  independent  of  the  technical  solution  alternatives? 

6.  Does  the  set  of  MOEs  address  any  required  KPPs? 

-  MOPs  - 

1.  Are  the  MOPs  traceable  to  the  applicable  MOEs,  KPPs,  system-level  performance 
requirements,  and  risks? 

2.  Do  the  MOPs  focus  on  technical  risks  and  support  trades  of  alternative  solutions  at  the 
system  level? 

3.  Do  the  MOPs  collectively  provide  insight  into  the  system  affordability? 

4.  Have  MOPs  been  decomposed,  budgeted,  and  allocated  to  the  system  elements  that 
satisfy  them? 

5.  Are  there  MOP  reporting  mechanisms  for  each  MOP  that  show  progress  toward  meeting 
MOPs  through  aggregation  of  applicable  TPMs  and  projections  analyzed  against 
thresholds  at  which  corrective  actions  are  to  be  taken? 

6.  Have  prototypes,  simulations  or  models  been  developed  to  support  analysis  of  MOPs, 
where  warranted  to  provide  necessary  insight?  Have  these  activities  been  incorporated 
into  the  project  planning  and  execution? 

7.  Has  each  MOP  been  assigned  to  an  “owner”  or  responsible  individual/IPT  for  tracking, 
making  recommendations,  and  taking  action? 

8.  Asa  result  of  MOP  analysis,  is  a  revision  to  the  measure  warranted?  If  warranted,  has  the 
revision  been  analyzed  and  reviewed  by  the  appropriate  approval  authority  or  control 
board? 

-  TPMs  - 

1.  Are  the  TPMs  traceable  to  the  applicable  MOPs,  system  element  level  performance 
requirements/allocations,  quality  objectives,  risks,  and  WBS  elements? 
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2.  Are  there  TPM  reporting  meehanisms  for  eaeh  TPM  that  show  progress  toward  meeting 
TPMs,  and  thresholds  at  which  corrective  actions  are  to  be  taken? 

3.  Have  TPMs  been  further  decomposed,  budgeted,  and  allocated  to  lower  level  system 
elements  that  satisfy  them,  where  more  granularity  is  needed  based  on  risk? 

4.  Have  TPM  budgets  been  flowed  down  to  development  organizations,  procuring 
organizations  and  subcontractors? 

5.  Have  prototypes,  simulations  or  models  been  developed  to  support  analysis  of  TPMs, 
where  warranted  to  provide  necessary  insight?  Have  these  activities  been  incorporated 
into  the  project  planning  and  execution? 

6.  Has  each  TPM  been  assigned  to  an  “owner”  or  responsible  individual/IPT  for  tracking, 
making  recommendations,  and  taking  action? 

7.  Have  the  sources  of  the  TPM  measures  been  identified  and  the  processes  generating  those 
measures  instrumented  in  such  a  manner  as  to  collect  the  required  data. 

8.  Are  TPMs  integrated  into  the  project  scheduling  process?  Can  any  TPM  risks  be  retired 
early?  Is  there  sufficient  schedule/budget  to  deal  with  TPM  issues  late  in  the  project? 

9.  As  a  result  of  TPM  analysis,  is  a  revision  to  the  measure  warranted?  If  warranted,  has  the 
revision  been  analyzed  and  reviewed  by  the  appropriate  approval  authority  or  control 
board? 
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9  Application  Of  Technical  Measurement  In  Integrated 
Product  Teams  (IPTs) 

An  initial  set  of  KPPs,  MOEs,  MOPs,  and  TPMs  is  generally  seleeted  and  defined  by  both  the 
major  and  support  IPTs  (Integrated  Product  Teams),  with  both  acquirer  and  supplier 
concurrence,  early  in  the  life  cycle.  Each  IPT  is  responsible  for  selecting  and  defining  the 
measures  that  fall  within  their  area  of  responsibility.  This  set  is  then  monitored  routinely  by  the 
same  IPTs  during  the  life  cycle  execution.  A  subset  of  these  measures  are  critical  to  managing 
the  system  integration  risks.  These  are  selected  with  concurrence  of  the  System  Integration  IPT 
and  are  monitored  routinely  by  them.  TPMs  selected  should  be  those  required  to  track  IPT 
success,  and  therefore  should  be  a  part  of  the  IPT's  normal,  scheduled  work.  The  IPT  is 
responsible  for  defining  and  documenting  the  TPMs  that  are  applicable  to  its  scope  of  work.  The 
documentation  includes  the  specifications  of  the  measures,  as  well  as  TPM  profile  and  status 
charts. 

The  responsible  IPT  develops  the  TPM  Profile/Status  charts  (i.e.,  "tracking"  charts),  monitors 
and  assesses  the  status  on  a  continuous  basis,  focuses  efforts  on  problem  areas,  reports  on  the 
status  (during  regularly  scheduled  IPT  meetings,  project  management  meetings,  and  reviews), 
maintains  the  records  for  these  parameters,  and  provides  updated  TPM  values  to  the  IPT 
responsible  for  overall  system  performance,  as  well  as  other  relevant  IPTs.  TPM  values  that  are 
outliers  from  plan  or  exhibit  negative  trends  will  receive  attention  by  at  least  the  originating  IPT, 
and  if  deemed  necessary,  by  its  parent  IPT,  and  possibly  the  project  management  team.  It  is  the 
responsibility  of  the  associated  IPT  to  work  upward  through  the  IPT  structure  to  obtain  the 
proper  management  visibility,  resources,  and  actions  until  the  situation  is  resolved.  IPTs  provide 
TPM  status  and  explanations  when  needed.  Each  subcontractor's  TPM  values  should  be  reported 
through  its  managing  IPT,  at  the  same  reporting  frequency  as  those  of  the  managing  IPT. 

The  IPT  responsible  for  the  overall  system  performance  usually  administers  the  TPM  program, 
coordinating  the  TPM  work  of  all  IPTs,  and  integrating  this  work  with  the  technical  risk 
management  effort.  This  includes  maintaining  a  master  repository  and  aggregating  TPMs  to 
higher-level  TPMs,  MOPs,  and  MOEs  for  top-level  analysis  and  reporting. 


10  Candidate  Technical  Measures  Matrix  -  Preliminary  Draft 

Table  10-1  is  a  preliminary  draft  of  a  matrix  of  candidate  technical  measures.  It  reflects  a 
compilation  of  typical  measures  for  many  application  domains.  It  is  by  no  means  a 
comprehensive  list  of  all  possible  technical  measures,  since  any  technical  requirement  could 
have  an  associated  measure. 
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Information 

Category 

Measurable 

Concept 

Typical  TPM 
Examples 

Domain 

Applicability 

Notes 

Product  Size 
and  Stability 

Physical  Size 
and  Stability 

Useful  Life 

Broad 

*  Also  relates  to  functional  correctness  concept 

*  Could  be  system,  element,  or  component  (e.g. 
battery,  propulsion) 

Weight 

Moderate 

*  Weight  under  various  conditions  including  weight 
budget 

*  Of  the  system  and/or  it's  payload 

Volumetric  Capacity 

Moderate 

*  E.g.  gas,  oil,  propellant,  coolant,  air,  liquid,  air 
exchange 

*  Of  the  system  and/or  it's  payload 

Power 

Moderate 

*  Includes  capacity,  budget,  and  margin 

*  For  fuel,  battery,  etc. 

Structural  Load 

Moderate 

E.g.,  weight,  bearing,  and  capacity 

Links  /  Connections 

Moderate 

E.g.  virtual  or  physical,  quantity  of  or  concurrent 
connections 

Dimensions 

Moderate 

E.g.  height,  length,  width,  depth,  perimeter, 
circumference 

Launch  Area 

Limited 

Primarily  for  aircraft,  spacecraft,  missile  domains 

Coverage  Area 

Limited 

E.g.  Antenna,  sensor 

Beam  Width 

Limited 

E.g.  of  signal  transmission 

Product  Quality 

Functional 

Correctness 

Accuracy 

Broad 

E.g.  target,  point  position,  point  estimation,  tracking, 
data  transmission,  clock,  time  tag 

Operational 

Environment 

Characteristics 

Broad 

E.g.  operating  and  storage  temperature,  thermal 
limits,  vibration,  shock,  humidity,  nuclear 

Power  Performance 

Moderate 

E.g.  conditioning,  distribution,  range,  peak 

Frequency 

Moderate 

Includes  frequency,  minimum  and  maximums, 
operational  bandwidth 

Velocity  and 

Acceleration 

Moderate 

Maximum,  sustained,  penetration,  forgiven 
conditions,  etc. 

Emissions 

Moderate 

Includes  infrared,  electromagnetic,  radio-frequency, 
nuclear  (radiation),  etc. 

Interference 

Moderate 

Includes  electromagnetic  interference  (EMI),  Radio¬ 
frequency  interference  (RFI),  and  others 

Gain  /  Noise 

Performance 

Limited 

Includes  phase  noise,  receive,  transmit,  spurious, 
extraneous,  AM/FM/PM 
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Information 

Category 

Measurable 

Concept 

Typical  TPM 
Examples 

Domain 

Applicability 

Notes 

Kill  /  Escape  Efficiency 

Limited 

Includes  probability  of  escaping  hit 

Take-off  and  Landing 
Distance 

Limited 

*  Under  various  conditions 

*  Primarily  for  aircraft  and  spacecraft  (only  shuttle 
for  landing)  domains 

Altitude 

Limited 

Maximum,  minimum,  etc. 

Roll  and  Pitch 

Limited 

Range  to  Target 

Limited 

Drag 

Limited 

Thrust 

Limited 

Under  various  conditions 

Nuclear  Hardness 

Limited 

Includes  the  ability  to  operate  under  specified  levels 
of  radiation 

Supportability  - 
Maintainability 

Maintainability 

Characteristics 

Moderate 

May  include  component  isolation,  test  point  visibility 

Maintenance  Cost 

Moderate 

Could  also  be  under  Resources  and  Cost 
information  category,  however,  for  the  TPM  it  is 
used  as  a  factor  in  maintenance  strategy  trades  and 
total  ownership  cost  decisions 

Turnaround  Time 

Limited 

Time  necessary  to  complete  an  action,  return  the 
system  to  operational  status 

Reconfiguration  Time 

Moderate 

Time  to  reconfigure  systems  (e.g.,  multi-purpose) 
from  one  state/capability/site  configuration  to 
another 

Efficiency 

Utilization 

Broad 

*  Includes  CPU,  memory,  disk  I/O,  bus 

*  Includes  both  standard,  peak,  degraded  modes 

Response  Time 

Broad 

Includes  time  and  rate  for  response,  reaction, 
refresh,  display  refresh,  database  performance 

System  Cycle  Time  and 
Rates 

Broad 

*  Can  include  component  parts  including  set-up, 
initialization,  execution,  incremental  or  end-to-end 
processing  time  (run-time  efficiency),  database 
transaction  processing,  launch  rates,  firing  rates, 
etc. 

*  E.g.  time  to  process  one  new  target,  a  set  of 
database  actions,  restart  time,  cold  restart  time 

Throughput 

Moderate 

*  Includes  CPU,  memory,  disk  I/O,  bus 

*  Includes  both  standard,  peak,  degraded  modes 

Power  /  Fuel 
Consumption 

Moderate 

Link  /  Connection 

Timing 

Moderate 

Includes  budgets 
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Information 

Category 

Measurable 

Concept 

Typical  TPM 
Examples 

Domain 

Applicability 

Notes 

Cooling  Efficiency 

Moderate 

Includes  flow,  capacity,  heating,  dissipation 

Signal  Efficiency 

Limited 

Includes  signal  switching  time,  receiver  signal 
sensitivity,  signal  latency,  etc. 

Output  Power 

Limited 

Includes  power,  current,  voltage 

Portability 

Database  Scalability 

Limited 

Usability 

Operational  Productivity 

Broad 

Operator  Response 

Time 

Broad 

Operator  Errors 

Broad 

Ride  Quality 

Limited 

Dependability  - 
Reliability 

Mean  Time  To  Failure 

Broad 

*  Can  be  "critical"  failures 

*  Can  be  time  "to"  or  time  "between" 

*  Can  include  time  to  malfunction  (including  power 
loss) 

Failure  /  Fault 
Characteristics 

Broad 

Includes  rates,  detection  efficiency,  tolerance,  false 
alarms,  bit  error  rates,  error  budgets 

Availability  /  Downtime 

Broad 

*  Includes  operational  availability,  link/connection 
availability  among  many  others 

*  Downtime  includes  total,  scheduled  (preventive 
maintenance),  from  failures,  etc. 

Mean  Time  to  Restore 
System 

Moderate 

*  Average  time  to  return  system  to  full  operational 
capacity  following  a  failure 

*  Includes  saturation  recovery  time 
(telecommunication  domain) 

*  Restart  times  may  be  elements  of  this  measure 

Mean  Time  to  Repair 

Moderate 

average  time  to  fix  a  failure 

Reliability  Figure  of 

Merit 

Moderate 

Technology 

Effectiveness 

Technology 

Suitability 

Technology  Readiness 
Levels 

Broad 

Technology  Readiness  Levels  (TRL)  are  a 
systematic  metric/measurement  system  that 
supports  assessments  of  the  maturity  of  a  particular 
technology  and  the  consistent  comparison  of 
maturity  between  different  types  of  technology. 

See  Section  10  for  more  detail. 

Table  10-1  Candidate  Technical  Measures 
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11  Technology  Readiness  Levels 

A  concept  related  to  TPMs  is  the  Teehnology  Readiness  Levels  (TRLs),  originally  defined  by 
NASA.  The  NASA  website  defines  TRLs  as:  “Teehnology  Readiness  Levels  (TRL)  are  a 
systematie  metrie/measurement  system  that  supports  assessments  of  the  maturity  of  a  partieular 
teehnology  and  the  eonsistent  eomparison  of  maturity  between  different  types  of  teehnology.” 

One  of  the  major  faetors  affeeting  the  defined  toleranee  and  risk  bands  of  the  TPM  is  the 
maturity  of  the  teehnology  ineorporated  into  the  design  of  the  teehnieal  solution.  “TRLs 
deseribe  the  maturity  of  the  teehnology  being  used  to  meet  requirements  for  whieh  the  TPMs  are 
indieators.  Just  as  progress  on  TPMs  is  a  preeursor  to  projeet/management  sueeess  (as  indieated 
by  earned  value),  TRLs  are  a  preeursor  or  indieator  of  potential  diffieulty  in  meeting  teehnieal 
performanee.  A  projeet  plan  is  based  on  knowledge.  The  less  eertain  the  knowledge  (low 
teehnology  maturity)  and  more  risky  the  performanee  (diffieult  to  meet  requirements  refleeted 
through  the  TPMs)  the  more  likely  the  projeet  plan  will  show  stress  (re-plan,  re-budget,  re- 

o 

sehedule).  The  earlier  this  is  known,  the  less  likely  the  need  for  re-baseline.” 

Table  11-1  is  ineluded  as  referenee  information.  It  is  derived  from  eharts  prepared  by  Mike 
Ferraro  of  the  Defense  Contraet  Management  Ageney,  provided  in  a  briefing  entitled 
“DoD/NDIA  Systems  Engineering  Committee  Meeting:  Teehnieal  Performanee  Measurement” 
and  dated  12  February  2002. 

A  GAO  Report  showed  teehnologies  introdueed  at  TRLs  of  5  or  lower  eneountered  maturation 
diffieulties  and  eontributed  to  problems  in  produet  development.  Those  produets  whose 
teehnologies  reaehed  high  TRLs  when  they  were  introdueed  were  better  able  to  meet  eost, 
sehedule,  and  performanee  requirements.  As  eaeh  sueeeeding  level  of  readiness  is  demonstrated, 
unknowns  are  replaeed  by  knowledge  and  gap  is  redueed. 

Additional  information  about  TRLs  ean  be  found  at  the  following  NASA  website: 
http://www.ase.nasa.gov/aboutus/trl-mtroduetion.html 


“  Ferraro  presentation 
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Technology  Readiness  Levels 

Descriptions 

1.  Basic  principles  observed  and  reported. 

Lowest  level  of  technology  readiness.  Scientific  research 
begins  to  be  translated  into  applied  research  and 
development.  Examples  might  include  paper  studies  of  a 
technology’s  basic  properties. 

2.  Technology  concept  and/or  application  formulated. 

Invention  begins.  Once  basic  principles  are  observed, 
practical  applications  can  be  invented.  The  application  is 
speculative  and  there  is  no  proof  or  detailed  analysis  to 
support  the  assumption.  Examples  are  still  limited  to 
paper  studies. 

3.  Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept. 

Active  research  and  development  is  initiated.  This 
includes  analytical  and  laboratory  studies  to  physically 
validate  analytical  predictions  of  separate  elements  of 
the  technology.  Examples  include  components  that  are 
not  yet  integrated  or  representative. 

4.  Component  and/or  breadboard  validation  in  laboratory 
environment. 

Basic  technological  components  are  integrated  to 
establish  that  the  pieces  will  work  together.  This  is 
relatively  “low  fidelity”  compared  to  the  eventual  system. 
Examples  include  integration  of  “ad  hoc”  hardware  in  a 
laboratory. 

5.  Component  and/or  breadboard  validation  in  relevant 
environment. 

Fidelity  of  breadboard  technology  increases  significantly. 
The  basic  technological  components  are  integrated  with 
reasonably  realistic  supporting  elements  so  that  the 
technology  can  be  tested  in  a  simulated  environment. 
Examples  include  “high  fidelity”  laboratory  integration  of 
components. 

6.  System/subsystem  model  or  prototype  demonstration 
in  a  relevant  environment. 

Representative  model  or  prototype  system,  which  is  well 
beyond  the  breadboard  tested  for  TRL  5,  is  tested  in  a 
relevant  environment.  Represents  a  major  step  up  in  a 
technology’s  demonstrated  readiness.  Examples  include 
testing  a  prototype  in  a  high  fidelity  laboratory 
environment  or  in  simulated  operational  environment. 

7.  System  prototype  demonstration  in  an  operational 
environment. 

Prototype  near  or  at  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6,  requiring  the 
demonstration  of  an  actual  system  prototype  in  an 
operational  environment,  such  as  an  aircraft,  vehicle  or 
space.  Examples  include  testing  the  prototype  in  a  test 
bed  aircraft. 

8.  Actual  system  completed  and  “flight  qualified”  through 
test  and  demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and 
under  expected  conditions.  In  almost  all  cases,  this  TRL 
represents  the  end  of  true  system  development. 

Examples  include  development  test  and  evaluation  of  the 
system  in  its  intended  weapon  system  to  determine  if  it 
meets  design  specifications. 

9.  Actual  system  “flight  proven”  through  successful 
mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and 
under  mission  conditions,  such  as  those  encountered  in 
operational  test  and  evaluation.  In  almost  all  cases,  this 
is  the  end  of  the  last  “bug  fixing”  aspects  of  true  system 
development.  Examples  include  using  the  system  under 
operational  mission  conditions. 

Table  11-1  Technology  Readiness  Level  Descriptions 
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12  Demographics  of  Questionnaire  Respondents 

The  Technical  Measurement  Questionnaire  (TMQ)  was  distributed  in  2002  to  members  of  the 
PSM  Technical  Working  Group  (TWG),  INCOSE  Technical  Committees  (TCs),  and  to  various 
other  interested  members  of  industry  and  government.  There  were  3 1  valid  questionnaire 
respondents.  The  information  in  this  section  provides  analysis  of  respondents’  demographics 
and  existing  usage  of  technical  measures. 

12.1  Respondent  Demographics 

The  respondent  group  was  very  experienced  in  the  area  of  technical  measurement  with  nearly 
80%  of  the  respondents  having  more  than  5  years  experience.  Figure  12-1  shows  that  only  10% 
of  the  respondents  had  2  or  less  years  of  experience  with  technical  measurement. 


Years  of  Experience  With  Technical  Measures 


Figure  12-1  Years  of  Experience  With  Technical  Measures 

Figure  12-2  shows  the  distribution  of  the  disciplines  of  the  respondents.  This  shows  that  the 
majority  of  the  respondents  were  from  the  areas  of  systems  and  software  engineering  with  some 
representation  of  quality  management/assurance  and  configuration  management.  None  of  the 
respondents  represented  hardware  engineering  or  product  support  disciplines. 

With  respect  to  their  business  areas,  the  respondents  were  spread  across  aerospace,  civil  aircraft, 
defense,  energy  and  environment,  information  and  data  systems,  and  information  technology. 
There  was  a  larger  number  of  respondents  from  aerospace  and  defense. 

Upon  examining  the  roles  of  the  respondents,  it  was  noted  that  there  was  a  wide  range  of 
technical  management  roles  represented.  The  roles  included  company  president  and  vice 
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president,  technical/engineering  fellows,  engineering  directors/managers,  principal/lead 
engineers,  engineering  process  project  directors/chairs,  quality  assurance  staff,  and  measurement 
coordinators/staff.  The  group  of  respondents  has  significant  experience  in  technical 
measurement  and  are  primarily  in  technical  leadership  and  management  roles  in  their 
organizations. 


Figure  12-2  Engineering  Disciplines  of  the  TMQ  Respondents 

12.2  Usage  Of  MOEs,  MOPs,  And  TPMs 

MOEs,  MOPs  and  TPMs  were  all  used  by  the  respondents,  but  in  varying  degrees.  Figure  12-3 
shows  the  results  of  questions  regarding  usage  of  these  measures.  Over  two-thirds  of  the 
respondents  indicated  use  of  MOEs,  half  of  the  respondents  use  MOPs,  and  nearly  all  use  TPMs. 


Usage  of  MOEs,  MOPs  &  TPMs 
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Figure  12-3  Usage  of  MOEs,  MOPs,  and  TPMs 

From  reading  the  text  responses,  it  is  apparent  that  some  organizations  or  projects  use  a  hierarchy 
of  TPMs  that  include  what  would  otherwise  be  considered  MOPs  and  possibly  MOEs.  This 
may  account  for  the  significantly  larger  percent  of  respondents  indicating  use  of  TPMs  than  the 
other  types  of  measures.  In  addition,  MOEs  are  not  often  passed  down  to  subcontractors.  The 
results  indicated  that  when  MOPs  are  used,  MOEs  are  also  used.  However,  the  opposite  is  not 
true. 
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13  Example  of  Compatible  Set  of  MOEs,  MOPs,  and  TPMs 

13.1  Overview/Relationships  Among  MOEs,  MOPs,  and  TPMs 

This  section  provides  an  example  to  illustrate  the  concept.  The  example  pertains  to  the 
development  of  a  military  aircraft.  It  is  useful  to  think  of  MOEs,  MOPs,  and  TPMs  as  an 
hierarchy  as  follows.  “Mission”  related  measures  of  success  (MOEs)  are  developed  by  the 
customer,  user,  or  acquirer,  in  terms  of  what  needs  to  be  achieved  within  the  operating 
environment  (i.e.,  the  primary  mission  objectives).  MOEs  are  defined  in  terms  of  the  effects  on 
the  mission.  The  MOEs  are  then  translated  into  or  elaborated  in  the  MOPs.  The  MOPs  describe 
the  system  that  is  to  be  built  to  meet  the  mission  objectives.  This  description  is  given  in  terms  of 
its  physical  attributes  (how  big,  etc.)  and  functional  attributes  (what  the  system  will  do  when 
built  and  operational).  The  MOPs  for  the  system  are  elaborated  into  very  specific  measures,  the 
TPMs,  that  can  be  used  during  the  course  of  the  project  to  determine  if  the  system  appears  likely 
to  realize  its  requirements  or  not.  There  can  be  a  current  estimate  for  the  final  value  of  a  TPM, 
such  as  the  Manufacturer’s  Empty  Weight  (MEW)  for  an  aircraft  (see  later  in  the  section  for 
more  details  of  the  MEW).  Thus,  there  would  be  a  progression  of  estimates  for  the  final  value  of 
the  MEW  during  the  course  of  the  development.  If  it  appears  unlikely  that  the  MEW  objective 
will  be  achieved,  then  action  should  be  taken  to  reduce  the  probability  that  this  will  be  the  case 
when  the  development  of  the  aircraft  has  been  completed. 

MOEs  characterize  a  system  in  terms  of  what  it  is  supposed  to  do,  its  “mission,”  not  how  the 
mission  is  to  be  accomplished.  However,  they  may  include  various  limitations  on  the  system. 
MOEs  constitute  the  highest  level  of  requirements  for  the  system  of  interest.  MOPs  deal  with  the 
hows  at  the  system  level,  various  aspects  of  implementation  of  the  mission.  TPMs  include 
specific  measurable  aspects  of  the  implementation  of  the  system  elements  that  can  be  measured 
(not  estimated  or  projected)  on  an  ongoing  basis  during  the  development  cycle.  By  “measurable” 
we  mean  both  directly  measurable  quantities,  such  as  the  weight  of  an  element  of  an  aircraft,  the 
number  of  code  statements  in  a  software  system  as  well  as  indirectly  measurable  quantities,  ones 
that  are  based  on  measurable  quantities,  such  as  the  mean-time -between- failure  of  a  jet  engine 
turbine  blade,  or  of  a  unit  of  software  code. 

TPMs  are  measures  at  a  sufficiently  detailed  level  that  they  can  be  used  during  a  system 
development  project  to  assess  progress  in  developing  the  system  and  to  enhance  the  degree  of 
confidence  that  the  system,  when  completed,  will  achieve  objectives  of  the  system  reflected  by 
the  MOPs,  will  meet  the  criteria  of  mission  success  given  by  the  MOEs.  TPMs  are  selected  so 
that  they  parse  or  decompose  the  high  level  requirements  imposed  upon  the  system  by  the  MOEs 
and  MOPs  to  relate  to  system  element  requirements.  TPMs  can  be  used  as  part  of  a  closed  loop 
quantitative  management  process  at  the  level  of  the  organization(s)  responsible  for  meeting  the 
goal  values  for  the  TPM(s). 
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13.2  Comprehensive  Example 

This  sub-section  provides  a  somewhat  simplified  example  of  a  sub-set  of  MOEs,  MOPs,  and 
TPMs  that  might  be  used  in  connection  with  the  development  of  a  new  fighter/attack  aircraft. 

Two  principal  top  level  mission  requirements  for  this  aircraft  are  given.  These  requirements  are 
the  basis  of  MOEs. 

Mission  Requirement  1 ;  The  aircraft  has  to  be  able  to  deliver  4000  pounds  of  ordnance  to  targets 
at  a  distance  of  up  to  300  nautical  miles  from  the  point  of  takeoff  and  then  return  to  the  point  of 
takeoff  without  being  refueled. 

Mission  Requirement  2;  The  aircraft  has  to  be  able  to  remain  on  station  in  the  vicinity  of  the 
target  for  up  to  14  hour  and  conduct  evasive  maneuvers  Ifom  possible  enemy  aircraft  and  missiles 
continuously  during  that  time. 

Operational  objectives  that  characterize  the  mission;  1)  Ability  to  operate  on  short  runways  (a 
particular  minimum  number  of  feet  would  be  specified  in  an  actual  project);  2)  Ability  to  be 
operated  with  a  crew  of  one. 

Eigure  13-1  presents  an  example  hierarchy  of  MOEs,  MOPs,  and  TPMs.  The  MOE  is  the 
requirement  for  the  aircraft  to  be  able  to  carry  4000#  of  ordnance  to  its  target.  The  MOP  is  the 
Overall  Aircraft  Weight  (or  operational  weight)  that  consists  of  1  fixed  portion  and  3  variable 
portions.  The  TPM  is  the  MEW  (Manufacturer’s  Empty  Weight),  which  is  the  fixed  portion  of 
the  weight.  Additionally,  the  MEW  would  likely  be  further  allocated  to  lower  level  TPMs  for 
the  System  Element  Weights  (per  the  allocation  given  in  the  weight  budget).  The  allocated 
weights  and  associated  TPMs  may  each  be  the  responsibility  of  a  separate  IPT. 


Figure  13-1  MOE,  MOP,  and  TPM  Hierarchy 
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Note  that  each  of  these  requirements,  e.g.,  ability  to  deliver  4000  pounds  of  ordnance,  probably 
has  a  degree  of  “fuzziness”  or  flexibility  (i.e.,  range  of  acceptability  that  can  be  used  in  trades). 
This  degree  of  flexibility  can  be  stated  formally  for  each  MOE  by  plotting  a  range  of  its  values 
versus  the  utilities  associated  with  them.  The  utility  is  a  measure  of  the  value  to  the  acquirer 
associated  with  a  particular  value  of  the  MOE.  Eor  example,  4000#  might  have  a  utility  of  1.00; 
i.e.,  it  meets  requirements.  But,  a  carrying  capacity  of  4500#  might  have  a  utility  of  1.20;  it 
substantially  exceeds  requirements,  and  might  provide  “extra  credit”  to  the  supplier  if  it  could  be 
achieved  without  sacrificing  the  attainment  of  any  of  the  other  MOEs  or  adding  cost/schedule. 
However,  a  capacity  of  3500#  might  have  a  utility  of  0.80;  it  substantially  underruns  the 
ordnance  capacity  requirement,  but  may  be  suitable,  when  the  degrees  of  satisfaction  of  other 
MOEs  and  resources  are  taken  into  account.  However,  a  carrying  capacity  of  3000#  might  have 
a  utility  of  0.00;  i.e.,  it  is  a  completely  unsatisfactory  value.  The  carrying  capacity  case, 
hypothetically,  would  look  like  this  as  a  degradation  of  500#  from  the  intended  4000#  figure 
would  mean  that  one  less  500#  bomb  could  be  carried  than  desired,  but  it  would  be  minimally 
satisfactory  under  certain  conditions  of  a  tradeoff,  say  between  aircraft  weight  and  therefore  fuel 
capacity  and  ordnance  carrying  and  delivery  capacity.  Figure  13-2  portrays  the  range  of  possible 
ordnance  capacity  values  and  their  respective  utilities. 


The  overall  aircraft  weight  is  a  key  MOP  that  is  now  considered.  The  development  of  a  new 
aircraft  would  typically  have  many  lower-level  TPMs  associated  with  a  budget  for  the  overall 
aircraft  weight.  An  aircraft’s  overall  (or  operational)  weight  is  the  sum  of  a  fixed  part  and  a 
variable  part.  The  fixed  part  is  the  Manufacturer’s  Empty  Weight,  MEW.  The  variable  part  is 
the  sum  of  the  weights  of  the  crew,  fuel,  ordnance,  etc.,  which  can  be  traded  to  some  extent  to 
provide  flexibility  in  the  mission.  The  values  for  these  TPMs  could  be  used  during  the  course  of 
the  development  of  the  aircraft  to  provide  detailed  ongoing  insight  into  progress  and  thus  the 
likelihood  of  meeting  the  MOP  (overall  aircraft  weight).  The  target  value  for  overall  aircraft 
weight  is  selected  in  part  based  on  its  relationship  with  various  measures,  such  as  jet  engine 
thrust  and  fuel  bum  rate.  The  TPM  (MEW)  planned  value  is  the  result  of  allocation  from  the 
MOP  (overall  aircraft  weight).  Since  the  MEW  is  the  fixed  weight,  the  planned  value  can  be 
tracked  throughout  the  development  process.  If  the  projected  MEW  value  at  the  completion  of 
the  development  process  is  anticipated  to  exceed  the  goal  established  for  it,  action  can  be  taken 
to  try  to  reduce  its  actual  value  (i.e.,  the  MEW  upon  completion  of  the  aircraft’s  development). 
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An  expectation  for  the  range  of  possible  variation  from  the  planned  value  at  various  points  in  the 
development  process  might  be  obtained  based  on  past  experience  in  the  development  of  other 
aircraft.  That  experience  can  be  used  as  part  of  the  quantitative  management  process  to  help 
anticipate  problems  in  attaining  the  MEW  planned  value,  i.e.,  possible  “weight  growth.”  On  the 
basis  of  this  past  experience,  the  developers  might  take  some  action  to  avert  growth  or  increase 
in  the  estimated  actual  MEW  as  the  air  vehicle  is  put  together.  The  objectives  of  reducing  likely 
MEW  growth  relative  to  the  desired  value  could  be  viewed  as  a  process  improvement 
opportunity.  Eurther,  this  information  could  be  used  in  connection  with  trade-off  studies  in  which 
the  air  vehicle’s  weight  is  traded  off  against  its  capacity  to  carry  ordnance,  its  prospective 
operational  range,  on  target  loitering  time,  and  take  off  weight  limits.  Based  on  experience  with 
other  aircraft  developments  that  the  development  team  might  have,  they  could  construct  a  range 
of  values  for  MEW  that  might  be  anticipated  to  be  estimated  or  projected  during  the 
development  process,  relative  to  the  planned  value.  An  example  is  shown  in  Eigure  13-3. 

In  this  example,  the  planned  MEW  is  25000#  and  the  aircraft  development  time  is  60  months. 

The  data  depicted  in  the  figure  would  be  constructed,  based  on  actual  experiential  weight  and 
development  schedule  data  from  prior  aircraft  developments  that  is  normalized  (i.e.,  MEW 
values  as  percent  of  planned  value  and  schedule  as  percent  of  development  effort  completed). 

The  data  in  the  figure  suggests  that  problems  might  be  expected  in  achieving  the  desired  MEW 
value,  as  has  been  the  case  for  past  projects.  These  curves  mirror  experience  in  achieving  cost 
objectives  as  well.  Eigure  13-4  provides  the  utilities  or  values  to  the  project  of  achieving  a  range 
of  possible  outcomes  for  MEW.  This  information  can  be  used,  all  other  things  being  equal,  to 
anticipate  likely  difficulties  in  achieving  the  MEW  objective  and  taking  some  action  to  either 
relax  the  goal  (i.e.,  increase  the  acceptable  MEW  value)  or  to  take  action  to  ensure  that  it  is 
achieved.  Of  course,  trade-off  studies  might  show  that  the  desired  value  of  MEW  cannot  be 
achieved  if  the  air  vehicle  structure  is  to  be  able  to  accommodate  the  amount  of  ordnance  that  it 
must  carry  and  the  amount  of  fuel  that  it  must  carry  in  order  to  enable  the  aircraft  to  reach  its 
target,  loiter  there  a  bit,  and  return  to  base  without  in  flight  refueling  being  necessary. 

Conducting  such  trade  studies  is  not  a  simple  matter.  However,  being  to  able  quantify  and 
normalize  past  experience  can  be  extremely  useful  in  the  conduct  of  a  new  aircraft  development 
project.  Indeed,  past  experience  is  a  good  indicator  of  what  might  well  happen  in  the  new 
instance,  absent  any  improvements  in  the  aircraft  development  process. 


Figure  13-3  Possible  MEW  Growth  vs.  Time  During  Aircraft  Development 


27  December  2005 


62 


Technical  Measurement 


INCOSE-TP-2003-020-01 


1.00 

d) 

3 

s 

23000  23500  24C 

300  24500  25000  25J 

500  26C 

MBA 

CD 

CM 

O 

O 

500  27C 

300  27; 

500  28C 

300  28; 

500 

Figure  13-4  Utilities  For  A  Range  of  Possible  MEW  Values 


13.3  Trade-offs  and  Programmatic  Objectives 

Based  on  insight  gained  from  the  measures  (e.g.,  MOPs  and  TPMs),  trade-offs  might  need  to  be 
made  amongst  two  or  more  requirements.  For  example,  the  mission  requirements  to  deliver 
4000#  of  ordnance  up  to  300  nautical  miles  might  be  have  traded  off  based  on  actual  measures 
obtained  during  the  development  of  the  aircraft.  For  example,  it  might  have  been  determined  that 
the  allocation  of  capacity  for  carrying  fuel  would  be  insufficient  for  the  aircraft  to  carry  4000#  a 
distance  of  300  nautical  miles  and  then  return  to  its  place  of  mission  departure.  Other  trade-offs 
might  have  to  be  made  that  deal  with  measures  outside  the  set  {MOEs,  MOPs,  TPMs}.  There 
might  be  trade-offs  made  between  programmatic  objectives,  such  as  schedule  and  development 
cost  and  some  requirement,  such  as  one  relating  to  a  defensive  weapons  targeting  capability. 
During  the  development  of  the  aircraft,  it  might  be  determined  that  the  likely  value  of  MEW 
exceeds  the  planned  value  such  that  the  maximum  mission  range  would  have  to  be  reduced  from 
the  planned  value  in  the  mission  requirement  (i.e.,  number  one)  that  states  it.  In  this  case,  a 
choice/trade-off  might  be  considered.  One  alternative  would  be  to  spend  enough  time  in  the 
development  process  to  determine  how  to  reduce  the  (prospective)  MEW  so  that  its  planned 
value  can  be  attained  and  extending  the  schedule  an  amount  of  time  sufficient  to  do  so.  Another 
alternative  would  be  to  meet  schedule  and  not  meet  the  MEW  planned  value  in  the  initial 
delivery  of  aircraft. 
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15  Acronyms 

AHP 

Analytical  Hierarchy  Process 

CI 

Configuration  Item 

CPU 

Computer  Processing  Unit 

CTC 

Critical  to  Customer 

EMI 

Electromagnetic  Interference 

EOC 

End  of  Contract 

EVMS 

Earned  Value  Management  System 

IPTs 

Integrated  Product  Teams 

KPPs 

Key  Performance  Parameters 

MOEs 

Measures  of  Effectiveness 

MOPs 

Measures  of  Performance 

MOS 

Measures  of  Suitability 

RFI 

Radio-frequency  Interference 

TCs 

Technical  Committees 

TMQ 

Technical  Measurement  Questionnaire 

TPMs 

Technical  Performance  Measures 

TRL 

Technical  Readiness  Eevel 

WBS 

Work  Breakdown  Structure 

27  December  2005 


65 


